2024-04-24T00:24:06.971 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T00:24:06.972 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535 --name teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} -- /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535/orig.config.yaml 2024-04-24T00:24:07.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T00:24:07.098 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535 branch: squid description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} email: ceph-qa@ceph.com first_in_suite: false job_id: '7670535' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9514 sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 sleep_before_teardown: 0 subset: 69/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b98aab6c00f483888c049d0d8e7bc507e115dd86/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9+zocXEZ6/Q4H0ypnly+tswU5kASmcq3gkzceQm1BjozrYH+u6sbPUXdJsrPoFMUTI0zrUdlvUpdEYYKYMGJE= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK0J1VHusBSk22A7Iy4CLA6n9LTzYYGkZ0qzQ4HEt7zMSj/jgAQFKkF1oxXDprXTf+LUXSk/vsF+zE0l0tfD1W4= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: pool RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: journal RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T00:24:07.098 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b98aab6c00f483888c049d0d8e7bc507e115dd86/qa; will attempt to use it 2024-04-24T00:24:07.099 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b98aab6c00f483888c049d0d8e7bc507e115dd86/qa/tasks 2024-04-24T00:24:07.099 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T00:24:07.100 INFO:teuthology.task.internal:Checking packages... 2024-04-24T00:24:07.124 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b98aab6c00f483888c049d0d8e7bc507e115dd86' 2024-04-24T00:24:07.124 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T00:24:07.124 INFO:teuthology.packaging:ref: None 2024-04-24T00:24:07.124 INFO:teuthology.packaging:tag: None 2024-04-24T00:24:07.124 INFO:teuthology.packaging:branch: squid 2024-04-24T00:24:07.124 INFO:teuthology.packaging:sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:24:07.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-24T00:24:07.527 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:24:07.528 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T00:24:07.535 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T00:24:07.535 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T00:24:07.578 INFO:teuthology.task.internal:Saving configuration 2024-04-24T00:24:07.596 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T00:24:07.612 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T00:24:07.637 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 00:18:31.330045', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMfqaUUtNxN6sbrl728kHPqiZPSVCK8TAkLOV64/rJgr'} 2024-04-24T00:24:07.659 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 00:18:31.331416', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIELJ/i1KOmCt0S55TagV9XTYj8S5nAgyyh+y54fACkPR'} 2024-04-24T00:24:07.659 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T00:24:07.665 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-04-24T00:24:07.665 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-04-24T00:24:07.665 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T00:24:07.769 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f084784b310>, signals=[15]) 2024-04-24T00:24:07.769 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T00:24:07.777 INFO:teuthology.task.internal:Opening connections... 2024-04-24T00:24:07.777 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T00:24:07.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:24:07.862 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T00:24:07.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:24:07.939 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T00:24:07.946 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-04-24T00:24:07.953 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-04-24T00:24:07.953 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-04-24T00:24:07.999 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-24T00:24:07.999 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Ubuntu" 2024-04-24T00:24:07.999 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="22.04" 2024-04-24T00:24:07.999 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-24T00:24:07.999 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_CODENAME=jammy 2024-04-24T00:24:07.999 INFO:teuthology.orchestra.run.smithi067.stdout:ID=ubuntu 2024-04-24T00:24:07.999 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE=debian 2024-04-24T00:24:08.000 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T00:24:08.000 INFO:teuthology.orchestra.run.smithi067.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T00:24:08.000 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T00:24:08.000 INFO:teuthology.orchestra.run.smithi067.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T00:24:08.000 INFO:teuthology.orchestra.run.smithi067.stdout:UBUNTU_CODENAME=jammy 2024-04-24T00:24:08.000 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-04-24T00:24:08.021 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-04-24T00:24:08.026 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-04-24T00:24:08.026 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-04-24T00:24:08.073 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-24T00:24:08.073 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="Ubuntu" 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="22.04" 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_CODENAME=jammy 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:ID=ubuntu 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE=debian 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T00:24:08.074 INFO:teuthology.orchestra.run.smithi134.stdout:UBUNTU_CODENAME=jammy 2024-04-24T00:24:08.074 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-04-24T00:24:08.096 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T00:24:08.106 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T00:24:08.128 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T00:24:08.128 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-04-24T00:24:08.131 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-04-24T00:24:08.135 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T00:24:08.154 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T00:24:08.154 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-04-24T00:24:08.176 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-04-24T00:24:08.191 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T00:24:08.242 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T00:24:08.256 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T00:24:08.257 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T00:24:08.257 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T00:24:08.257 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T00:24:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:24:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:24:08.263 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-24T00:24:08.263 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-24T00:24:08.310 INFO:teuthology.orchestra.run.smithi134.stdout:5.15.0-56-generic 2024-04-24T00:24:08.310 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.15.0-56-generic 2024-04-24T00:24:08.310 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get clean 2024-04-24T00:24:08.312 INFO:teuthology.orchestra.run.smithi067.stdout:5.15.0-56-generic 2024-04-24T00:24:08.312 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.15.0-56-generic 2024-04-24T00:24:08.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-04-24T00:24:08.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get update 2024-04-24T00:24:08.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-04-24T00:24:08.590 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:24:08.590 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-24T00:24:08.591 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:24:08.593 INFO:teuthology.orchestra.run.smithi134.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-24T00:24:08.595 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-24T00:24:08.599 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-24T00:24:08.682 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-24T00:24:08.690 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-24T00:24:08.819 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,392 kB] 2024-04-24T00:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,392 kB] 2024-04-24T00:24:08.890 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-24T00:24:08.910 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-24T00:24:08.937 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-24T00:24:08.943 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-24T00:24:08.948 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-24T00:24:08.948 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,768 kB] 2024-04-24T00:24:08.955 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-24T00:24:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-24T00:24:08.966 INFO:teuthology.orchestra.run.smithi134.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-24T00:24:08.967 INFO:teuthology.orchestra.run.smithi134.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,768 kB] 2024-04-24T00:24:08.976 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-24T00:24:08.986 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-24T00:24:08.987 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB] 2024-04-24T00:24:08.990 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T00:24:08.991 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-24T00:24:09.006 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-24T00:24:09.022 INFO:teuthology.orchestra.run.smithi134.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-24T00:24:09.023 INFO:teuthology.orchestra.run.smithi134.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB] 2024-04-24T00:24:09.029 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-24T00:24:09.029 INFO:teuthology.orchestra.run.smithi134.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T00:24:09.030 INFO:teuthology.orchestra.run.smithi134.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-24T00:24:09.031 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-24T00:24:09.046 INFO:teuthology.orchestra.run.smithi134.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-24T00:24:09.055 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-24T00:24:09.056 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-24T00:24:09.069 INFO:teuthology.orchestra.run.smithi134.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-24T00:24:09.070 INFO:teuthology.orchestra.run.smithi134.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-24T00:24:09.071 INFO:teuthology.orchestra.run.smithi134.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-24T00:24:09.082 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-24T00:24:09.082 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-24T00:24:09.092 INFO:teuthology.orchestra.run.smithi134.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-24T00:24:09.093 INFO:teuthology.orchestra.run.smithi134.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-24T00:24:09.097 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-24T00:24:09.101 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-24T00:24:09.101 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-24T00:24:09.102 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-24T00:24:09.107 INFO:teuthology.orchestra.run.smithi134.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-24T00:24:09.107 INFO:teuthology.orchestra.run.smithi134.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-24T00:24:09.109 INFO:teuthology.orchestra.run.smithi134.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-24T00:24:09.132 INFO:teuthology.orchestra.run.smithi134.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-24T00:24:09.133 INFO:teuthology.orchestra.run.smithi134.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-24T00:24:09.134 INFO:teuthology.orchestra.run.smithi134.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-24T00:24:09.138 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-24T00:24:09.150 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T00:24:09.150 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-24T00:24:09.163 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-24T00:24:09.189 INFO:teuthology.orchestra.run.smithi134.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-24T00:24:09.189 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-24T00:24:09.193 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-24T00:24:09.193 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-24T00:24:09.194 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-24T00:24:09.195 INFO:teuthology.orchestra.run.smithi134.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T00:24:09.195 INFO:teuthology.orchestra.run.smithi134.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-24T00:24:09.195 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-24T00:24:09.209 INFO:teuthology.orchestra.run.smithi134.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-24T00:24:09.212 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-24T00:24:09.212 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-24T00:24:09.213 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-24T00:24:09.214 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-24T00:24:09.226 INFO:teuthology.orchestra.run.smithi134.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-24T00:24:09.230 INFO:teuthology.orchestra.run.smithi134.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-24T00:24:09.230 INFO:teuthology.orchestra.run.smithi134.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-24T00:24:09.231 INFO:teuthology.orchestra.run.smithi134.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-24T00:24:09.231 INFO:teuthology.orchestra.run.smithi134.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-24T00:24:09.232 INFO:teuthology.orchestra.run.smithi134.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-24T00:24:09.235 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-24T00:24:09.235 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-24T00:24:09.236 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-24T00:24:09.236 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-24T00:24:09.236 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-24T00:24:09.251 INFO:teuthology.orchestra.run.smithi134.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-24T00:24:09.253 INFO:teuthology.orchestra.run.smithi134.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-24T00:24:09.254 INFO:teuthology.orchestra.run.smithi134.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-24T00:24:09.254 INFO:teuthology.orchestra.run.smithi134.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-24T00:24:09.273 INFO:teuthology.orchestra.run.smithi134.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-24T00:24:09.273 INFO:teuthology.orchestra.run.smithi134.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-24T00:24:09.274 INFO:teuthology.orchestra.run.smithi134.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-24T00:24:09.274 INFO:teuthology.orchestra.run.smithi134.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-24T00:24:14.653 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 13.2 MB in 2s (7,134 kB/s) 2024-04-24T00:24:15.046 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 13.2 MB in 2s (6,780 kB/s) 2024-04-24T00:24:15.961 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-24T00:24:15.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T00:24:16.040 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-24T00:24:16.233 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2024-04-24T00:24:16.234 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2024-04-24T00:24:16.405 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-24T00:24:16.410 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:24:16.410 INFO:teuthology.orchestra.run.smithi134.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:24:16.411 INFO:teuthology.orchestra.run.smithi134.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:24:16.411 INFO:teuthology.orchestra.run.smithi134.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:24:16.411 INFO:teuthology.orchestra.run.smithi134.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-24T00:24:16.411 INFO:teuthology.orchestra.run.smithi134.stdout: python2.7-minimal 2024-04-24T00:24:16.411 INFO:teuthology.orchestra.run.smithi134.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:24:16.413 INFO:teuthology.orchestra.run.smithi134.stdout:The following additional packages will be installed: 2024-04-24T00:24:16.413 INFO:teuthology.orchestra.run.smithi134.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T00:24:16.413 INFO:teuthology.orchestra.run.smithi134.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-24T00:24:16.413 INFO:teuthology.orchestra.run.smithi134.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-24T00:24:16.414 INFO:teuthology.orchestra.run.smithi134.stdout:Suggested packages: 2024-04-24T00:24:16.415 INFO:teuthology.orchestra.run.smithi134.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-24T00:24:16.415 INFO:teuthology.orchestra.run.smithi134.stdout:Recommended packages: 2024-04-24T00:24:16.415 INFO:teuthology.orchestra.run.smithi134.stdout: thermald 2024-04-24T00:24:16.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T00:24:16.463 INFO:teuthology.orchestra.run.smithi134.stdout:The following NEW packages will be installed: 2024-04-24T00:24:16.463 INFO:teuthology.orchestra.run.smithi134.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T00:24:16.463 INFO:teuthology.orchestra.run.smithi134.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-24T00:24:16.463 INFO:teuthology.orchestra.run.smithi134.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-24T00:24:16.464 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages will be upgraded: 2024-04-24T00:24:16.465 INFO:teuthology.orchestra.run.smithi134.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T00:24:16.489 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-24T00:24:16.645 INFO:teuthology.orchestra.run.smithi134.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T00:24:16.645 INFO:teuthology.orchestra.run.smithi134.stdout:Need to get 113 MB of archives. 2024-04-24T00:24:16.645 INFO:teuthology.orchestra.run.smithi134.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-24T00:24:16.645 INFO:teuthology.orchestra.run.smithi134.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-24T00:24:16.696 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-04-24T00:24:16.696 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-04-24T00:24:16.880 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:24:16.880 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:24:16.880 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:24:16.881 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:24:16.881 INFO:teuthology.orchestra.run.smithi067.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-24T00:24:16.881 INFO:teuthology.orchestra.run.smithi067.stdout: python2.7-minimal 2024-04-24T00:24:16.881 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:24:16.882 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-04-24T00:24:16.882 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T00:24:16.882 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-24T00:24:16.883 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-24T00:24:16.884 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-04-24T00:24:16.884 INFO:teuthology.orchestra.run.smithi067.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-24T00:24:16.884 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-04-24T00:24:16.884 INFO:teuthology.orchestra.run.smithi067.stdout: thermald 2024-04-24T00:24:16.931 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-04-24T00:24:16.931 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T00:24:16.931 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-24T00:24:16.931 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-24T00:24:16.933 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages will be upgraded: 2024-04-24T00:24:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T00:24:17.136 INFO:teuthology.orchestra.run.smithi067.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T00:24:17.136 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 113 MB of archives. 2024-04-24T00:24:17.136 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-24T00:24:17.136 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-24T00:24:18.060 INFO:teuthology.orchestra.run.smithi134.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-24T00:24:18.408 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-24T00:24:18.534 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-24T00:24:18.941 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-24T00:24:20.558 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-24T00:24:20.559 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-24T00:24:20.559 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-24T00:24:21.135 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-24T00:24:21.135 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-24T00:24:21.135 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-24T00:24:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-24T00:24:21.596 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-24T00:24:21.666 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-24T00:24:21.714 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-24T00:24:22.079 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 113 MB in 5s (24.0 MB/s) 2024-04-24T00:24:22.099 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 113 MB in 5s (21.6 MB/s) 2024-04-24T00:24:22.268 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-24T00:24:22.296 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-24T00:24:24.199 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-24T00:24:24.203 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:24:24.222 INFO:teuthology.orchestra.run.smithi134.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-24T00:24:24.227 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:24:24.307 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:24.317 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:26.410 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-24T00:24:26.431 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:24:26.514 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:26.850 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-24T00:24:26.871 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:24:26.946 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-24T00:24:26.963 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:26.967 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:24:26.996 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:27.386 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-24T00:24:27.407 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:24:27.453 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:33.629 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:24:33.781 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:24:34.089 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:24:34.136 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:24:34.184 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:24:34.263 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:24:34.479 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-24T00:24:34.502 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-24T00:24:34.545 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T00:24:34.620 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:24:34.725 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:24:35.019 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-24T00:24:35.041 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-24T00:24:35.086 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T00:24:40.360 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-24T00:24:40.398 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:24:40.443 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:41.259 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-24T00:24:41.296 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:24:41.342 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:43.173 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:24:43.306 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:24:43.632 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T00:24:43.752 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:43.870 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-24T00:24:43.999 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:44.013 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:24:44.139 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:24:45.256 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T00:24:45.394 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:45.520 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-24T00:24:45.649 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:46.142 INFO:teuthology.orchestra.run.smithi067.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:24:46.142 INFO:teuthology.orchestra.run.smithi067.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-24T00:24:46.142 INFO:teuthology.orchestra.run.smithi067.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:24:46.142 INFO:teuthology.orchestra.run.smithi067.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-24T00:24:46.326 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:47.008 INFO:teuthology.orchestra.run.smithi134.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:24:47.008 INFO:teuthology.orchestra.run.smithi134.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-24T00:24:47.008 INFO:teuthology.orchestra.run.smithi134.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:24:47.008 INFO:teuthology.orchestra.run.smithi134.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-24T00:24:47.168 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:47.803 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:48.562 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:49.670 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-24T00:24:49.796 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-24T00:24:49.929 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-24T00:24:50.289 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:50.597 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-24T00:24:50.704 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T00:24:50.704 INFO:teuthology.orchestra.run.smithi067.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:24:50.957 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:24:51.005 INFO:teuthology.orchestra.run.smithi134.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T00:24:51.005 INFO:teuthology.orchestra.run.smithi134.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:25:04.916 INFO:teuthology.orchestra.run.smithi134.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T00:25:04.916 INFO:teuthology.orchestra.run.smithi134.stdout:Sourcing file `/etc/default/grub' 2024-04-24T00:25:04.941 INFO:teuthology.orchestra.run.smithi134.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T00:25:05.003 INFO:teuthology.orchestra.run.smithi134.stdout:Generating grub configuration file ... 2024-04-24T00:25:05.060 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T00:25:05.060 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub' 2024-04-24T00:25:05.080 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T00:25:05.106 INFO:teuthology.orchestra.run.smithi067.stdout:Generating grub configuration file ... 2024-04-24T00:25:05.276 INFO:teuthology.orchestra.run.smithi134.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:25:05.306 INFO:teuthology.orchestra.run.smithi134.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:25:05.375 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:25:05.399 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:25:05.674 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:25:05.680 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T00:25:05.683 INFO:teuthology.orchestra.run.smithi134.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:25:05.689 INFO:teuthology.orchestra.run.smithi134.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T00:25:05.752 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T00:25:05.757 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T00:25:05.759 INFO:teuthology.orchestra.run.smithi134.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T00:25:05.764 INFO:teuthology.orchestra.run.smithi134.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T00:25:05.929 INFO:teuthology.orchestra.run.smithi067.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T00:25:05.929 INFO:teuthology.orchestra.run.smithi067.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T00:25:05.929 INFO:teuthology.orchestra.run.smithi067.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T00:25:05.937 INFO:teuthology.orchestra.run.smithi134.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T00:25:05.938 INFO:teuthology.orchestra.run.smithi134.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T00:25:05.938 INFO:teuthology.orchestra.run.smithi134.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T00:25:05.984 INFO:teuthology.orchestra.run.smithi067.stdout:done 2024-04-24T00:25:05.985 INFO:teuthology.orchestra.run.smithi134.stdout:done 2024-04-24T00:25:06.466 DEBUG:teuthology.orchestra.run.smithi134:> dpkg -s linux-image-generic 2024-04-24T00:25:06.484 INFO:teuthology.orchestra.run.smithi134.stdout:Package: linux-image-generic 2024-04-24T00:25:06.484 INFO:teuthology.orchestra.run.smithi134.stdout:Status: install ok installed 2024-04-24T00:25:06.484 INFO:teuthology.orchestra.run.smithi134.stdout:Priority: optional 2024-04-24T00:25:06.484 INFO:teuthology.orchestra.run.smithi134.stdout:Section: kernel 2024-04-24T00:25:06.484 INFO:teuthology.orchestra.run.smithi134.stdout:Installed-Size: 21 2024-04-24T00:25:06.484 INFO:teuthology.orchestra.run.smithi134.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T00:25:06.484 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: amd64 2024-04-24T00:25:06.484 INFO:teuthology.orchestra.run.smithi134.stdout:Source: linux-meta 2024-04-24T00:25:06.485 INFO:teuthology.orchestra.run.smithi134.stdout:Version: 5.15.0.105.102 2024-04-24T00:25:06.485 INFO:teuthology.orchestra.run.smithi134.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T00:25:06.485 INFO:teuthology.orchestra.run.smithi134.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T00:25:06.485 INFO:teuthology.orchestra.run.smithi134.stdout:Recommends: thermald 2024-04-24T00:25:06.485 INFO:teuthology.orchestra.run.smithi134.stdout:Description: Generic Linux kernel image 2024-04-24T00:25:06.485 INFO:teuthology.orchestra.run.smithi134.stdout: This package will always depend on the latest generic kernel image 2024-04-24T00:25:06.485 INFO:teuthology.orchestra.run.smithi134.stdout: available. 2024-04-24T00:25:06.485 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-24T00:25:06.485 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T00:25:06.486 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T00:25:06.486 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-04-24T00:25:06.486 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T00:25:06.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get clean 2024-04-24T00:25:06.493 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-04-24T00:25:06.515 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-04-24T00:25:06.515 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 21 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.15.0.105.102 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T00:25:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-04-24T00:25:06.517 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-04-24T00:25:06.517 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-04-24T00:25:06.517 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-04-24T00:25:06.517 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-24T00:25:06.517 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T00:25:06.518 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T00:25:06.518 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-04-24T00:25:06.518 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T00:25:06.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-04-24T00:25:06.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get update 2024-04-24T00:25:06.808 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:25:06.808 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T00:25:06.821 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T00:25:06.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-04-24T00:25:06.840 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T00:25:06.963 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T00:25:07.080 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:25:07.164 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T00:25:07.244 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T00:25:08.180 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-24T00:25:08.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T00:25:08.250 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-24T00:25:08.429 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2024-04-24T00:25:08.429 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2024-04-24T00:25:08.431 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-24T00:25:08.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T00:25:08.503 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-24T00:25:08.591 INFO:teuthology.orchestra.run.smithi134.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-24T00:25:08.591 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:25:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:25:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:25:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:25:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T00:25:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T00:25:08.591 INFO:teuthology.orchestra.run.smithi134.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:25:08.627 INFO:teuthology.orchestra.run.smithi134.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T00:25:08.629 DEBUG:teuthology.orchestra.run.smithi134:> dpkg -s linux-image-generic 2024-04-24T00:25:08.685 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-04-24T00:25:08.686 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Package: linux-image-generic 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Status: install ok installed 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Priority: optional 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Section: kernel 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Installed-Size: 21 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: amd64 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Source: linux-meta 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Version: 5.15.0.105.102 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T00:25:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T00:25:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:Recommends: thermald 2024-04-24T00:25:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:Description: Generic Linux kernel image 2024-04-24T00:25:08.693 INFO:teuthology.orchestra.run.smithi134.stdout: This package will always depend on the latest generic kernel image 2024-04-24T00:25:08.693 INFO:teuthology.orchestra.run.smithi134.stdout: available. 2024-04-24T00:25:08.693 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-04-24T00:25:08.737 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.r7I9CYFK3p 2024-04-24T00:25:08.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub/grub.cfg /tmp/tmp.r7I9CYFK3p 2024-04-24T00:25:08.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.r7I9CYFK3p 2024-04-24T00:25:08.847 INFO:teuthology.orchestra.run.smithi067.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-24T00:25:08.847 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:25:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:25:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:25:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:25:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T00:25:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T00:25:08.848 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:25:08.888 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T00:25:08.890 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-04-24T00:25:08.923 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.r7I9CYFK3p is 10KB 2024-04-24T00:25:08.935 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.r7I9CYFK3p 2024-04-24T00:25:08.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T00:25:08.946 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-04-24T00:25:08.946 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-04-24T00:25:08.946 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 21 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.15.0.105.102 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-04-24T00:25:08.947 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-04-24T00:25:08.948 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-04-24T00:25:08.994 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.vT9oGBsq6y 2024-04-24T00:25:08.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub/grub.cfg /tmp/tmp.vT9oGBsq6y 2024-04-24T00:25:08.996 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:25:08.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T00:25:08.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T00:25:09.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.vT9oGBsq6y 2024-04-24T00:25:09.072 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-24T00:25:09.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo update-grub 2024-04-24T00:25:09.167 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.vT9oGBsq6y is 10KB 2024-04-24T00:25:09.182 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.vT9oGBsq6y 2024-04-24T00:25:09.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T00:25:09.243 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:25:09.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T00:25:09.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T00:25:09.310 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-24T00:25:09.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo update-grub 2024-04-24T00:25:10.297 INFO:teuthology.orchestra.run.smithi134.stderr:Sourcing file `/etc/default/grub' 2024-04-24T00:25:10.298 INFO:teuthology.orchestra.run.smithi134.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T00:25:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-04-24T00:25:10.526 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub' 2024-04-24T00:25:10.527 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T00:25:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-04-24T00:25:10.569 INFO:teuthology.orchestra.run.smithi134.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:25:10.575 INFO:teuthology.orchestra.run.smithi134.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:25:10.715 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:25:10.721 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:25:10.884 INFO:teuthology.orchestra.run.smithi134.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:25:10.889 INFO:teuthology.orchestra.run.smithi134.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T00:25:10.931 INFO:teuthology.orchestra.run.smithi134.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T00:25:10.937 INFO:teuthology.orchestra.run.smithi134.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T00:25:10.938 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:25:10.943 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T00:25:10.984 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T00:25:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T00:25:11.053 INFO:teuthology.orchestra.run.smithi134.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T00:25:11.053 INFO:teuthology.orchestra.run.smithi134.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T00:25:11.053 INFO:teuthology.orchestra.run.smithi134.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T00:25:11.072 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-04-24T00:25:11.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-04-24T00:25:11.099 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T00:25:11.100 INFO:teuthology.orchestra.run.smithi067.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T00:25:11.100 INFO:teuthology.orchestra.run.smithi067.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T00:25:11.108 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-04-24T00:25:11.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-04-24T00:25:41.111 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-04-24T00:25:41.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T00:25:41.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:25:41.129 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-04-24T00:25:41.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-24T00:25:41.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:26:15.711 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-24T00:26:24.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-24T00:26:24.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:26:41.132 DEBUG:teuthology.orchestra.remote:timed out 2024-04-24T00:26:43.106 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-24T00:26:50.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T00:26:50.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:26:50.480 ERROR:teuthology.orchestra.connection:Error authenticating with smithi134.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2024-04-24T00:26:54.564 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-04-24T00:26:55.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-24T00:26:55.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:26:56.015 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T00:26:56.015 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-105-generic"... 2024-04-24T00:26:56.015 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-24T00:26:56.020 INFO:teuthology.orchestra.run.smithi134.stdout:5.15.0-105-generic 2024-04-24T00:26:56.021 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-24T00:26:56.021 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T00:26:56.021 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T00:26:56.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-24T00:26:57.022 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-04-24T00:26:57.022 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T00:26:57.161 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-04-24T00:26:57.203 DEBUG:teuthology.parallel:result is None 2024-04-24T00:27:11.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-24T00:27:11.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:27:11.373 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-24T00:27:12.022 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-24T00:27:12.022 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-105-generic"... 2024-04-24T00:27:12.022 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-24T00:27:12.068 INFO:teuthology.orchestra.run.smithi067.stdout:5.15.0-105-generic 2024-04-24T00:27:12.068 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-24T00:27:12.068 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T00:27:12.069 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T00:27:13.070 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-04-24T00:27:13.070 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T00:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-04-24T00:27:13.178 DEBUG:teuthology.parallel:result is None 2024-04-24T00:27:13.178 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T00:27:13.188 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T00:27:13.188 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T00:27:13.190 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T00:27:13.197 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T00:27:13.242 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T00:27:13.288 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T00:27:13.289 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T00:27:13.292 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T00:27:13.316 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T00:27:13.335 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T00:27:13.335 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T00:27:13.345 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T00:27:13.367 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T00:27:13.372 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T00:27:13.377 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T00:27:13.380 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T00:27:13.382 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T00:27:13.388 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T00:27:13.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T00:27:13.421 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T00:27:13.438 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T00:27:13.447 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T00:27:13.448 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T00:27:13.481 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T00:27:13.490 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T00:27:13.532 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T00:27:13.581 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:27:13.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T00:27:13.643 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T00:27:13.648 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T00:27:13.699 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:27:13.699 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T00:27:13.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-04-24T00:27:13.775 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-04-24T00:27:13.856 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T00:27:13.883 INFO:teuthology.task.internal:Starting timer... 2024-04-24T00:27:13.883 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T00:27:13.938 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T00:27:13.986 DEBUG:teuthology.task.selinux:Excluding smithi067: OS 'ubuntu' does not support SELinux 2024-04-24T00:27:13.986 DEBUG:teuthology.task.selinux:Excluding smithi134: OS 'ubuntu' does not support SELinux 2024-04-24T00:27:13.987 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T00:27:13.987 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-24T00:27:13.987 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T00:27:13.987 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T00:27:14.120 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T00:27:14.137 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T00:27:14.139 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T00:34:21.493 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-04-24T00:34:21.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-24T00:34:21.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:34:21.580 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-24T00:34:21.658 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-24T00:34:21.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T00:34:21.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:34:21.737 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-04-24T00:34:21.810 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T00:34:21.810 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T00:34:21.825 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T00:34:21.825 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T00:34:21.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T00:34:21.828 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T00:34:21.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T00:34:21.857 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-24T00:34:21.857 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: Command line: ntpd -gq 2024-04-24T00:34:21.858 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: ---------------------------------------------------- 2024-04-24T00:34:21.858 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: ntp-4 is maintained by Network Time Foundation, 2024-04-24T00:34:21.858 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-24T00:34:21.858 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: corporation. Support and training for ntp-4 are 2024-04-24T00:34:21.858 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: available at https://www.nwtime.org/support 2024-04-24T00:34:21.858 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: ---------------------------------------------------- 2024-04-24T00:34:21.859 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: proto: precision = 0.060 usec (-24) 2024-04-24T00:34:21.859 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: basedate set to 2022-02-04 2024-04-24T00:34:21.859 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: gps base set to 2022-02-06 (week 2196) 2024-04-24T00:34:21.859 INFO:teuthology.orchestra.run.smithi067.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T00:34:21.859 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T00:34:21.859 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T00:34:21.860 INFO:teuthology.orchestra.run.smithi067.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T00:34:21.860 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T00:34:21.861 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T00:34:21.861 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T00:34:21.861 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: Listen normally on 3 enp3s0f1 172.21.15.67:123 2024-04-24T00:34:21.861 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: Listen normally on 4 lo [::1]:123 2024-04-24T00:34:21.861 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:fc0b%5]:123 2024-04-24T00:34:21.861 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:21 ntpd[17230]: Listening on routing socket on fd #22 for interface updates 2024-04-24T00:34:21.879 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-24T00:34:21.879 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: Command line: ntpd -gq 2024-04-24T00:34:21.879 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: ---------------------------------------------------- 2024-04-24T00:34:21.880 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: ntp-4 is maintained by Network Time Foundation, 2024-04-24T00:34:21.880 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-24T00:34:21.880 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: corporation. Support and training for ntp-4 are 2024-04-24T00:34:21.880 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: available at https://www.nwtime.org/support 2024-04-24T00:34:21.880 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: ---------------------------------------------------- 2024-04-24T00:34:21.880 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: proto: precision = 0.044 usec (-24) 2024-04-24T00:34:21.881 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: basedate set to 2022-02-04 2024-04-24T00:34:21.881 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: gps base set to 2022-02-06 (week 2196) 2024-04-24T00:34:21.881 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T00:34:21.881 INFO:teuthology.orchestra.run.smithi134.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T00:34:21.881 INFO:teuthology.orchestra.run.smithi134.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T00:34:21.882 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T00:34:21.882 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T00:34:21.882 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T00:34:21.882 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T00:34:21.882 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: Listen normally on 3 enp3s0f1 172.21.15.134:123 2024-04-24T00:34:21.883 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: Listen normally on 4 lo [::1]:123 2024-04-24T00:34:21.883 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6de9%5]:123 2024-04-24T00:34:21.883 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:21 ntpd[17261]: Listening on routing socket on fd #22 for interface updates 2024-04-24T00:34:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:22 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:22 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:22 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:22 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:22 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:22 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:22 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:22 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:23.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:23 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:23.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:23 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:23.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:23 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:23.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:23 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:23.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:23 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:23.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:23 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:23.882 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:23 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:24.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:24 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:24.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:24 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:24.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:24 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:24.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:24 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:24.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:24 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:24.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:24 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:24.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:24 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:24.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:24 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:24.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:24 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:24.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:24 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:24.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:24 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:24.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:24 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:25.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:25 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:25.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:25 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:25.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:25 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:25.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:25 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:25.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:25 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:25.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:25 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:25.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:25.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:25.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:25 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:26 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:26.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:26 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:26.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:26 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:26.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:26 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:26.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:26 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:26.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:26 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:26.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:26 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:27 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:27 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:27 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:27 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:27 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:27 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:27 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:28 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:28.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:28 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:28.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:28 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:28.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:28 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:28.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:28 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:28.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:28 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:28.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:28 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:29.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:29 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:29.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:29 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:29.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:29 ntpd[17230]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:29.858 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:29 ntpd[17230]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:29.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:29 ntpd[17230]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:29.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:29 ntpd[17230]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:29.859 INFO:teuthology.orchestra.run.smithi067.stdout:24 Apr 00:34:29 ntpd[17230]: ntpd: time slew -0.001969 s 2024-04-24T00:34:29.859 INFO:teuthology.orchestra.run.smithi067.stdout:ntpd: time slew -0.001969s 2024-04-24T00:34:29.859 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:29 ntpd[17230]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T00:34:29.860 INFO:teuthology.orchestra.run.smithi067.stderr:24 Apr 00:34:29 ntpd[17230]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T00:34:29.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:29.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:29.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:29.880 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:29.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:34:29.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:34:29.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:34:29.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:34:29.881 INFO:teuthology.orchestra.run.smithi134.stdout:24 Apr 00:34:29 ntpd[17261]: ntpd: time slew -0.001560 s 2024-04-24T00:34:29.881 INFO:teuthology.orchestra.run.smithi134.stdout:ntpd: time slew -0.001560s 2024-04-24T00:34:29.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T00:34:29.881 INFO:teuthology.orchestra.run.smithi134.stderr:24 Apr 00:34:29 ntpd[17261]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T00:34:29.908 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T00:34:29.908 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-04-24T00:34:29.908 INFO:teuthology.orchestra.run.smithi067.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:34:29.908 INFO:teuthology.orchestra.run.smithi067.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:34:29.908 INFO:teuthology.orchestra.run.smithi067.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:34:29.908 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:34:29.959 INFO:teuthology.orchestra.run.smithi134.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T00:34:29.960 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================== 2024-04-24T00:34:29.960 INFO:teuthology.orchestra.run.smithi134.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:34:29.960 INFO:teuthology.orchestra.run.smithi134.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:34:29.960 INFO:teuthology.orchestra.run.smithi134.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:34:29.960 INFO:teuthology.orchestra.run.smithi134.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:34:29.960 INFO:teuthology.run_tasks:Running task install... 2024-04-24T00:34:29.975 DEBUG:teuthology.task.install:project ceph 2024-04-24T00:34:29.975 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}} 2024-04-24T00:34:29.975 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'} 2024-04-24T00:34:29.975 INFO:teuthology.task.install:Using flavor: default 2024-04-24T00:34:29.981 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T00:34:29.981 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-04-24T00:34:29.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-key list | grep Ceph 2024-04-24T00:34:29.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-key list | grep Ceph 2024-04-24T00:34:30.042 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-24T00:34:30.056 INFO:teuthology.orchestra.run.smithi134.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-24T00:34:30.072 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T00:34:30.072 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T00:34:30.073 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-04-24T00:34:30.073 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:34:30.084 INFO:teuthology.orchestra.run.smithi134.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T00:34:30.084 INFO:teuthology.orchestra.run.smithi134.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T00:34:30.084 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-04-24T00:34:30.084 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:34:30.219 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default/ 2024-04-24T00:34:30.219 INFO:teuthology.task.install.deb:Package version is 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:34:30.311 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default/ 2024-04-24T00:34:30.311 INFO:teuthology.task.install.deb:Package version is 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:34:30.395 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:34:30.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T00:34:30.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-04-24T00:34:30.422 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:34:30.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T00:34:30.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get update 2024-04-24T00:34:30.630 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy InRelease 2024-04-24T00:34:30.652 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-24T00:34:30.674 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-24T00:34:30.696 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-24T00:34:30.715 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:34:30.718 INFO:teuthology.orchestra.run.smithi134.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy InRelease 2024-04-24T00:34:30.723 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T00:34:30.734 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T00:34:30.734 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:34:30.736 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-24T00:34:30.754 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-24T00:34:30.781 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T00:34:30.791 INFO:teuthology.orchestra.run.smithi134.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-24T00:34:30.807 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T00:34:30.828 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-24T00:34:30.863 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T00:34:30.870 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-24T00:34:30.888 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T00:34:31.099 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 28.9 kB in 0s (60.3 kB/s) 2024-04-24T00:34:31.111 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 28.9 kB in 1s (56.8 kB/s) 2024-04-24T00:34:32.087 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-24T00:34:32.095 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-24T00:34:32.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2424-gb98aab6c-1jammy cephadm=19.0.0-2424-gb98aab6c-1jammy ceph-mds=19.0.0-2424-gb98aab6c-1jammy ceph-mgr=19.0.0-2424-gb98aab6c-1jammy ceph-common=19.0.0-2424-gb98aab6c-1jammy ceph-fuse=19.0.0-2424-gb98aab6c-1jammy ceph-test=19.0.0-2424-gb98aab6c-1jammy ceph-volume=19.0.0-2424-gb98aab6c-1jammy radosgw=19.0.0-2424-gb98aab6c-1jammy python3-rados=19.0.0-2424-gb98aab6c-1jammy python3-rgw=19.0.0-2424-gb98aab6c-1jammy python3-cephfs=19.0.0-2424-gb98aab6c-1jammy python3-rbd=19.0.0-2424-gb98aab6c-1jammy libcephfs2=19.0.0-2424-gb98aab6c-1jammy libcephfs-dev=19.0.0-2424-gb98aab6c-1jammy librados2=19.0.0-2424-gb98aab6c-1jammy librbd1=19.0.0-2424-gb98aab6c-1jammy rbd-fuse=19.0.0-2424-gb98aab6c-1jammy rbd-mirror=19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:34:32.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2424-gb98aab6c-1jammy cephadm=19.0.0-2424-gb98aab6c-1jammy ceph-mds=19.0.0-2424-gb98aab6c-1jammy ceph-mgr=19.0.0-2424-gb98aab6c-1jammy ceph-common=19.0.0-2424-gb98aab6c-1jammy ceph-fuse=19.0.0-2424-gb98aab6c-1jammy ceph-test=19.0.0-2424-gb98aab6c-1jammy ceph-volume=19.0.0-2424-gb98aab6c-1jammy radosgw=19.0.0-2424-gb98aab6c-1jammy python3-rados=19.0.0-2424-gb98aab6c-1jammy python3-rgw=19.0.0-2424-gb98aab6c-1jammy python3-cephfs=19.0.0-2424-gb98aab6c-1jammy python3-rbd=19.0.0-2424-gb98aab6c-1jammy libcephfs2=19.0.0-2424-gb98aab6c-1jammy libcephfs-dev=19.0.0-2424-gb98aab6c-1jammy librados2=19.0.0-2424-gb98aab6c-1jammy librbd1=19.0.0-2424-gb98aab6c-1jammy rbd-fuse=19.0.0-2424-gb98aab6c-1jammy rbd-mirror=19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:34:32.155 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-24T00:34:32.174 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-24T00:34:32.334 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2024-04-24T00:34:32.334 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2024-04-24T00:34:32.376 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-04-24T00:34:32.376 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-04-24T00:34:32.508 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:34:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:34:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:34:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:34:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T00:34:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T00:34:32.509 INFO:teuthology.orchestra.run.smithi134.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:34:32.509 INFO:teuthology.orchestra.run.smithi134.stdout:The following additional packages will be installed: 2024-04-24T00:34:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T00:34:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-24T00:34:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T00:34:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T00:34:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet zip 2024-04-24T00:34:32.512 INFO:teuthology.orchestra.run.smithi134.stdout:Suggested packages: 2024-04-24T00:34:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-24T00:34:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-24T00:34:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-24T00:34:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-24T00:34:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-24T00:34:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T00:34:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-24T00:34:32.513 INFO:teuthology.orchestra.run.smithi134.stdout:Recommended packages: 2024-04-24T00:34:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: btrfs-tools 2024-04-24T00:34:32.576 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:34:32.577 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:34:32.577 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:34:32.577 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:34:32.577 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T00:34:32.577 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T00:34:32.577 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:34:32.578 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-04-24T00:34:32.578 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T00:34:32.578 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-24T00:34:32.578 INFO:teuthology.orchestra.run.smithi067.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T00:34:32.579 INFO:teuthology.orchestra.run.smithi067.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T00:34:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T00:34:32.582 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-24T00:34:32.582 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-24T00:34:32.582 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T00:34:32.582 INFO:teuthology.orchestra.run.smithi067.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-24T00:34:32.582 INFO:teuthology.orchestra.run.smithi067.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-24T00:34:32.582 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-24T00:34:32.582 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-24T00:34:32.582 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet zip 2024-04-24T00:34:32.582 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-04-24T00:34:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-24T00:34:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-24T00:34:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-24T00:34:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-24T00:34:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-24T00:34:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T00:34:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-24T00:34:32.583 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-04-24T00:34:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: btrfs-tools 2024-04-24T00:34:32.612 INFO:teuthology.orchestra.run.smithi134.stdout:The following NEW packages will be installed: 2024-04-24T00:34:32.613 INFO:teuthology.orchestra.run.smithi134.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T00:34:32.613 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T00:34:32.613 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-24T00:34:32.613 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-24T00:34:32.613 INFO:teuthology.orchestra.run.smithi134.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-24T00:34:32.613 INFO:teuthology.orchestra.run.smithi134.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-24T00:34:32.614 INFO:teuthology.orchestra.run.smithi134.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-24T00:34:32.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-24T00:34:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-24T00:34:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-24T00:34:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-24T00:34:32.616 INFO:teuthology.orchestra.run.smithi134.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-04-24T00:34:32.685 INFO:teuthology.orchestra.run.smithi134.stdout:0 upgraded, 119 newly installed, 0 to remove and 302 not upgraded. 2024-04-24T00:34:32.685 INFO:teuthology.orchestra.run.smithi134.stdout:Need to get 183 MB of archives. 2024-04-24T00:34:32.685 INFO:teuthology.orchestra.run.smithi134.stdout:After this operation, 834 MB of additional disk space will be used. 2024-04-24T00:34:32.686 INFO:teuthology.orchestra.run.smithi134.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-24T00:34:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-04-24T00:34:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T00:34:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T00:34:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-24T00:34:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-24T00:34:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-24T00:34:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-24T00:34:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-24T00:34:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-24T00:34:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-24T00:34:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-24T00:34:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-24T00:34:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-24T00:34:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-04-24T00:34:32.758 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 119 newly installed, 0 to remove and 302 not upgraded. 2024-04-24T00:34:32.758 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 183 MB of archives. 2024-04-24T00:34:32.758 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 834 MB of additional disk space will be used. 2024-04-24T00:34:32.759 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-24T00:34:32.797 INFO:teuthology.orchestra.run.smithi134.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-24T00:34:32.801 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-24T00:34:32.822 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-24T00:34:32.831 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2424-gb98aab6c-1jammy [3,544 kB] 2024-04-24T00:34:32.871 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-24T00:34:32.876 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-24T00:34:32.897 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-24T00:34:32.903 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-24T00:34:32.911 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-24T00:34:32.918 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-24T00:34:32.923 INFO:teuthology.orchestra.run.smithi134.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-24T00:34:32.924 INFO:teuthology.orchestra.run.smithi134.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-24T00:34:32.925 INFO:teuthology.orchestra.run.smithi134.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-24T00:34:32.936 INFO:teuthology.orchestra.run.smithi134.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-24T00:34:32.939 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2424-gb98aab6c-1jammy [3,544 kB] 2024-04-24T00:34:32.939 INFO:teuthology.orchestra.run.smithi134.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-24T00:34:32.943 INFO:teuthology.orchestra.run.smithi134.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-24T00:34:32.946 INFO:teuthology.orchestra.run.smithi134.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-24T00:34:32.982 INFO:teuthology.orchestra.run.smithi134.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-24T00:34:32.986 INFO:teuthology.orchestra.run.smithi134.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi134.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi134.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi134.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi134.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi134.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi134.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi134.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi134.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi134.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi134.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi134.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi134.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi134.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi134.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi134.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi134.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi134.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi134.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-24T00:34:33.102 INFO:teuthology.orchestra.run.smithi134.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-24T00:34:33.104 INFO:teuthology.orchestra.run.smithi134.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-24T00:34:33.104 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-24T00:34:33.105 INFO:teuthology.orchestra.run.smithi134.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-24T00:34:33.115 INFO:teuthology.orchestra.run.smithi134.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-24T00:34:33.116 INFO:teuthology.orchestra.run.smithi134.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-24T00:34:33.117 INFO:teuthology.orchestra.run.smithi134.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-24T00:34:33.117 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-24T00:34:33.118 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-24T00:34:33.118 INFO:teuthology.orchestra.run.smithi134.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-24T00:34:33.120 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-24T00:34:33.131 INFO:teuthology.orchestra.run.smithi134.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-24T00:34:33.140 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-24T00:34:33.146 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-24T00:34:33.150 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-24T00:34:33.153 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-24T00:34:33.166 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-24T00:34:33.168 INFO:teuthology.orchestra.run.smithi134.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-24T00:34:33.169 INFO:teuthology.orchestra.run.smithi134.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-24T00:34:33.171 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-24T00:34:33.172 INFO:teuthology.orchestra.run.smithi134.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-24T00:34:33.174 INFO:teuthology.orchestra.run.smithi134.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-24T00:34:33.176 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-24T00:34:33.178 INFO:teuthology.orchestra.run.smithi134.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-24T00:34:33.179 INFO:teuthology.orchestra.run.smithi134.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-24T00:34:33.181 INFO:teuthology.orchestra.run.smithi134.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-24T00:34:33.186 INFO:teuthology.orchestra.run.smithi134.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-24T00:34:33.227 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-24T00:34:33.228 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-24T00:34:33.229 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-24T00:34:33.231 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-24T00:34:33.232 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-24T00:34:33.232 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-24T00:34:33.233 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-24T00:34:33.285 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-24T00:34:33.285 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-24T00:34:33.286 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-24T00:34:33.330 INFO:teuthology.orchestra.run.smithi134.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-24T00:34:33.331 INFO:teuthology.orchestra.run.smithi134.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-24T00:34:33.333 INFO:teuthology.orchestra.run.smithi134.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T00:34:33.333 INFO:teuthology.orchestra.run.smithi134.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-24T00:34:33.335 INFO:teuthology.orchestra.run.smithi134.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-24T00:34:33.338 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-24T00:34:33.339 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-24T00:34:33.340 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-24T00:34:33.341 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-24T00:34:33.343 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T00:34:33.343 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-24T00:34:33.344 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-24T00:34:33.379 INFO:teuthology.orchestra.run.smithi134.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-24T00:34:33.382 INFO:teuthology.orchestra.run.smithi134.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-24T00:34:33.383 INFO:teuthology.orchestra.run.smithi134.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-24T00:34:33.383 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-24T00:34:33.383 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-24T00:34:33.384 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-24T00:34:33.406 INFO:teuthology.orchestra.run.smithi134.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-24T00:34:33.406 INFO:teuthology.orchestra.run.smithi134.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-24T00:34:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-24T00:34:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-24T00:34:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-24T00:34:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-24T00:34:33.423 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-24T00:34:33.424 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-24T00:34:33.425 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-24T00:34:33.426 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-24T00:34:33.429 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-24T00:34:33.433 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-24T00:34:33.434 INFO:teuthology.orchestra.run.smithi067.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-24T00:34:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-24T00:34:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-24T00:34:33.453 INFO:teuthology.orchestra.run.smithi067.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-24T00:34:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-24T00:34:33.454 INFO:teuthology.orchestra.run.smithi067.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-24T00:34:33.459 INFO:teuthology.orchestra.run.smithi067.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-24T00:34:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-24T00:34:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-24T00:34:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-24T00:34:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-24T00:34:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-24T00:34:33.490 INFO:teuthology.orchestra.run.smithi067.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-24T00:34:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-24T00:34:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-24T00:34:33.494 INFO:teuthology.orchestra.run.smithi067.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-24T00:34:33.496 INFO:teuthology.orchestra.run.smithi067.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-24T00:34:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-24T00:34:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-24T00:34:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-24T00:34:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T00:34:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-24T00:34:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-24T00:34:33.590 INFO:teuthology.orchestra.run.smithi134.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-24T00:34:33.592 INFO:teuthology.orchestra.run.smithi134.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-24T00:34:33.609 INFO:teuthology.orchestra.run.smithi134.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-24T00:34:33.610 INFO:teuthology.orchestra.run.smithi134.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-24T00:34:33.632 INFO:teuthology.orchestra.run.smithi067.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-24T00:34:33.634 INFO:teuthology.orchestra.run.smithi067.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-24T00:34:33.636 INFO:teuthology.orchestra.run.smithi067.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T00:34:33.637 INFO:teuthology.orchestra.run.smithi067.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-24T00:34:33.639 INFO:teuthology.orchestra.run.smithi067.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi134.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi134.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi134.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-24T00:34:33.744 INFO:teuthology.orchestra.run.smithi067.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-24T00:34:33.748 INFO:teuthology.orchestra.run.smithi067.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-24T00:34:33.749 INFO:teuthology.orchestra.run.smithi067.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-24T00:34:33.785 INFO:teuthology.orchestra.run.smithi067.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-24T00:34:33.785 INFO:teuthology.orchestra.run.smithi067.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-24T00:34:33.786 INFO:teuthology.orchestra.run.smithi067.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-24T00:34:33.787 INFO:teuthology.orchestra.run.smithi067.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-24T00:34:33.788 INFO:teuthology.orchestra.run.smithi067.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-24T00:34:33.789 INFO:teuthology.orchestra.run.smithi067.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-24T00:34:33.789 INFO:teuthology.orchestra.run.smithi067.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-24T00:34:33.794 INFO:teuthology.orchestra.run.smithi067.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-24T00:34:33.805 INFO:teuthology.orchestra.run.smithi067.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-24T00:34:33.819 INFO:teuthology.orchestra.run.smithi067.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-24T00:34:33.822 INFO:teuthology.orchestra.run.smithi067.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-24T00:34:33.823 INFO:teuthology.orchestra.run.smithi067.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-24T00:34:33.832 INFO:teuthology.orchestra.run.smithi067.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-24T00:34:33.839 INFO:teuthology.orchestra.run.smithi067.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-24T00:34:33.841 INFO:teuthology.orchestra.run.smithi067.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-24T00:34:33.842 INFO:teuthology.orchestra.run.smithi067.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-24T00:34:33.852 INFO:teuthology.orchestra.run.smithi067.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-24T00:34:33.854 INFO:teuthology.orchestra.run.smithi067.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-24T00:34:33.864 INFO:teuthology.orchestra.run.smithi067.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-24T00:34:33.878 INFO:teuthology.orchestra.run.smithi067.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T00:34:33.880 INFO:teuthology.orchestra.run.smithi067.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-24T00:34:33.880 INFO:teuthology.orchestra.run.smithi067.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-24T00:34:33.885 INFO:teuthology.orchestra.run.smithi067.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-24T00:34:33.887 INFO:teuthology.orchestra.run.smithi067.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-24T00:34:33.907 INFO:teuthology.orchestra.run.smithi067.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-24T00:34:33.908 INFO:teuthology.orchestra.run.smithi067.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-24T00:34:33.909 INFO:teuthology.orchestra.run.smithi067.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-24T00:34:33.916 INFO:teuthology.orchestra.run.smithi067.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-24T00:34:33.918 INFO:teuthology.orchestra.run.smithi067.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-24T00:34:34.114 INFO:teuthology.orchestra.run.smithi134.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2424-gb98aab6c-1jammy [3,218 kB] 2024-04-24T00:34:34.774 INFO:teuthology.orchestra.run.smithi067.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2424-gb98aab6c-1jammy [3,218 kB] 2024-04-24T00:34:34.880 INFO:teuthology.orchestra.run.smithi134.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2424-gb98aab6c-1jammy [711 kB] 2024-04-24T00:34:34.999 INFO:teuthology.orchestra.run.smithi134.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2424-gb98aab6c-1jammy [356 kB] 2024-04-24T00:34:35.069 INFO:teuthology.orchestra.run.smithi134.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2424-gb98aab6c-1jammy [32.8 kB] 2024-04-24T00:34:35.070 INFO:teuthology.orchestra.run.smithi134.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2424-gb98aab6c-1jammy [180 kB] 2024-04-24T00:34:35.106 INFO:teuthology.orchestra.run.smithi134.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2424-gb98aab6c-1jammy [65.7 kB] 2024-04-24T00:34:35.108 INFO:teuthology.orchestra.run.smithi134.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2424-gb98aab6c-1jammy [330 kB] 2024-04-24T00:34:35.179 INFO:teuthology.orchestra.run.smithi134.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2424-gb98aab6c-1jammy [6,775 kB] 2024-04-24T00:34:35.711 INFO:teuthology.orchestra.run.smithi067.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2424-gb98aab6c-1jammy [711 kB] 2024-04-24T00:34:35.906 INFO:teuthology.orchestra.run.smithi067.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2424-gb98aab6c-1jammy [356 kB] 2024-04-24T00:34:36.012 INFO:teuthology.orchestra.run.smithi067.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2424-gb98aab6c-1jammy [32.8 kB] 2024-04-24T00:34:36.015 INFO:teuthology.orchestra.run.smithi067.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2424-gb98aab6c-1jammy [180 kB] 2024-04-24T00:34:36.087 INFO:teuthology.orchestra.run.smithi067.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2424-gb98aab6c-1jammy [65.7 kB] 2024-04-24T00:34:36.092 INFO:teuthology.orchestra.run.smithi067.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2424-gb98aab6c-1jammy [330 kB] 2024-04-24T00:34:36.213 INFO:teuthology.orchestra.run.smithi067.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2424-gb98aab6c-1jammy [6,775 kB] 2024-04-24T00:34:36.357 INFO:teuthology.orchestra.run.smithi134.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2424-gb98aab6c-1jammy [112 kB] 2024-04-24T00:34:36.384 INFO:teuthology.orchestra.run.smithi134.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2424-gb98aab6c-1jammy [466 kB] 2024-04-24T00:34:36.457 INFO:teuthology.orchestra.run.smithi134.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2424-gb98aab6c-1jammy [25.6 MB] 2024-04-24T00:34:37.867 INFO:teuthology.orchestra.run.smithi067.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2424-gb98aab6c-1jammy [112 kB] 2024-04-24T00:34:37.872 INFO:teuthology.orchestra.run.smithi067.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2424-gb98aab6c-1jammy [466 kB] 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi067.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2424-gb98aab6c-1jammy [25.6 MB] 2024-04-24T00:34:40.543 INFO:teuthology.orchestra.run.smithi134.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2424-gb98aab6c-1jammy [5,084 kB] 2024-04-24T00:34:41.130 INFO:teuthology.orchestra.run.smithi134.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2424-gb98aab6c-1jammy [244 kB] 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi134.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2424-gb98aab6c-1jammy [124 kB] 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi134.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2424-gb98aab6c-1jammy [1,526 kB] 2024-04-24T00:34:41.396 INFO:teuthology.orchestra.run.smithi134.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2424-gb98aab6c-1jammy [6,246 kB] 2024-04-24T00:34:42.130 INFO:teuthology.orchestra.run.smithi134.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2424-gb98aab6c-1jammy [22.7 MB] 2024-04-24T00:34:42.691 INFO:teuthology.orchestra.run.smithi067.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2424-gb98aab6c-1jammy [5,084 kB] 2024-04-24T00:34:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2424-gb98aab6c-1jammy [244 kB] 2024-04-24T00:34:43.506 INFO:teuthology.orchestra.run.smithi067.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2424-gb98aab6c-1jammy [124 kB] 2024-04-24T00:34:43.542 INFO:teuthology.orchestra.run.smithi067.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2424-gb98aab6c-1jammy [1,526 kB] 2024-04-24T00:34:43.820 INFO:teuthology.orchestra.run.smithi067.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2424-gb98aab6c-1jammy [6,246 kB] 2024-04-24T00:34:45.422 INFO:teuthology.orchestra.run.smithi067.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2424-gb98aab6c-1jammy [22.7 MB] 2024-04-24T00:34:46.631 INFO:teuthology.orchestra.run.smithi134.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2424-gb98aab6c-1jammy [14.0 kB] 2024-04-24T00:34:46.632 INFO:teuthology.orchestra.run.smithi134.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2424-gb98aab6c-1jammy [869 kB] 2024-04-24T00:34:46.672 INFO:teuthology.orchestra.run.smithi134.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2424-gb98aab6c-1jammy [2,437 kB] 2024-04-24T00:34:47.100 INFO:teuthology.orchestra.run.smithi134.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2424-gb98aab6c-1jammy [790 kB] 2024-04-24T00:34:47.246 INFO:teuthology.orchestra.run.smithi134.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2424-gb98aab6c-1jammy [147 kB] 2024-04-24T00:34:47.260 INFO:teuthology.orchestra.run.smithi134.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2424-gb98aab6c-1jammy [2,295 kB] 2024-04-24T00:34:47.654 INFO:teuthology.orchestra.run.smithi134.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2424-gb98aab6c-1jammy [8,625 kB] 2024-04-24T00:34:49.037 INFO:teuthology.orchestra.run.smithi067.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2424-gb98aab6c-1jammy [14.0 kB] 2024-04-24T00:34:49.038 INFO:teuthology.orchestra.run.smithi067.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2424-gb98aab6c-1jammy [869 kB] 2024-04-24T00:34:49.087 INFO:teuthology.orchestra.run.smithi067.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2424-gb98aab6c-1jammy [2,437 kB] 2024-04-24T00:34:49.194 INFO:teuthology.orchestra.run.smithi134.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2424-gb98aab6c-1jammy [14.2 kB] 2024-04-24T00:34:49.194 INFO:teuthology.orchestra.run.smithi134.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2424-gb98aab6c-1jammy [50.7 MB] 2024-04-24T00:34:49.399 INFO:teuthology.orchestra.run.smithi067.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2424-gb98aab6c-1jammy [790 kB] 2024-04-24T00:34:49.483 INFO:teuthology.orchestra.run.smithi067.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2424-gb98aab6c-1jammy [147 kB] 2024-04-24T00:34:49.508 INFO:teuthology.orchestra.run.smithi067.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2424-gb98aab6c-1jammy [2,295 kB] 2024-04-24T00:34:49.794 INFO:teuthology.orchestra.run.smithi067.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2424-gb98aab6c-1jammy [8,625 kB] 2024-04-24T00:34:50.830 INFO:teuthology.orchestra.run.smithi067.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2424-gb98aab6c-1jammy [14.2 kB] 2024-04-24T00:34:50.840 INFO:teuthology.orchestra.run.smithi067.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2424-gb98aab6c-1jammy [50.7 MB] 2024-04-24T00:34:56.168 INFO:teuthology.orchestra.run.smithi134.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2424-gb98aab6c-1jammy [121 kB] 2024-04-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi134.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2424-gb98aab6c-1jammy [38.9 kB] 2024-04-24T00:34:56.172 INFO:teuthology.orchestra.run.smithi134.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2424-gb98aab6c-1jammy [13.4 MB] 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi067.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2424-gb98aab6c-1jammy [121 kB] 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi067.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2424-gb98aab6c-1jammy [38.9 kB] 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi067.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2424-gb98aab6c-1jammy [13.4 MB] 2024-04-24T00:34:58.176 INFO:teuthology.orchestra.run.smithi134.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2424-gb98aab6c-1jammy [91.4 kB] 2024-04-24T00:34:58.178 INFO:teuthology.orchestra.run.smithi134.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-2424-gb98aab6c-1jammy [3,122 kB] 2024-04-24T00:34:59.050 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 183 MB in 26s (7,061 kB/s) 2024-04-24T00:34:59.239 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-24T00:34:59.340 INFO:teuthology.orchestra.run.smithi134.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-24T00:34:59.346 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-24T00:34:59.438 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T00:34:59.758 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T00:34:59.784 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-24T00:34:59.824 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T00:35:00.169 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T00:35:00.195 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T00:35:00.431 INFO:teuthology.orchestra.run.smithi067.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2424-gb98aab6c-1jammy [91.4 kB] 2024-04-24T00:35:00.433 INFO:teuthology.orchestra.run.smithi067.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-2424-gb98aab6c-1jammy [3,122 kB] 2024-04-24T00:35:00.673 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T00:35:00.699 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:35:00.755 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:35:01.109 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T00:35:01.127 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:35:01.158 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:35:01.487 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T00:35:01.502 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:35:01.545 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:35:01.880 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 183 MB in 29s (6,386 kB/s) 2024-04-24T00:35:01.898 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-24T00:35:01.924 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-24T00:35:01.965 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T00:35:02.040 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-24T00:35:02.144 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-24T00:35:02.150 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-24T00:35:02.239 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T00:35:02.251 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librados2. 2024-04-24T00:35:02.277 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../007-librados2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:02.309 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librados2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:02.592 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T00:35:02.619 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-24T00:35:02.658 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T00:35:02.745 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librbd1. 2024-04-24T00:35:02.772 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../008-librbd1_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:02.812 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librbd1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:02.945 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T00:35:02.972 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-24T00:35:03.003 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T00:35:03.298 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T00:35:03.325 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:03.365 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libcephfs2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T00:35:03.475 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:35:03.523 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:35:03.650 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rados. 2024-04-24T00:35:03.666 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:03.700 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rados (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:03.967 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T00:35:03.969 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T00:35:03.994 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:35:03.995 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:04.028 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-ceph-argparse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:04.034 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:35:04.272 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T00:35:04.287 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:04.455 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cephfs (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:04.495 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T00:35:04.522 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:35:04.553 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:35:04.725 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:04.849 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-24T00:35:04.875 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-24T00:35:04.915 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T00:35:05.061 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T00:35:05.076 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-24T00:35:05.111 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T00:35:05.234 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librados2. 2024-04-24T00:35:05.261 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../007-librados2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:05.301 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librados2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:05.405 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T00:35:05.432 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-24T00:35:05.472 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-24T00:35:05.724 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T00:35:05.740 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:05.753 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librbd1. 2024-04-24T00:35:05.780 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../008-librbd1_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:05.782 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rbd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:05.820 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librbd1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:06.828 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T00:35:06.854 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-24T00:35:06.903 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T00:35:07.065 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T00:35:07.081 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:07.115 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:07.163 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-24T00:35:07.190 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-24T00:35:07.230 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T00:35:07.375 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rados. 2024-04-24T00:35:07.391 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:07.434 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rados (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:07.533 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-24T00:35:07.548 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-24T00:35:07.591 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T00:35:07.753 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T00:35:07.780 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:07.819 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-argparse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:07.911 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package lua5.1. 2024-04-24T00:35:07.937 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-24T00:35:07.994 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-24T00:35:08.072 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T00:35:08.088 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:08.122 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cephfs (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:08.282 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package lua-any. 2024-04-24T00:35:08.297 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-24T00:35:08.340 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-24T00:35:08.416 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T00:35:08.443 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:08.483 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:08.593 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package zip. 2024-04-24T00:35:08.608 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-24T00:35:08.651 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking zip (3.0-12build2) ... 2024-04-24T00:35:08.794 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T00:35:08.821 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-24T00:35:08.861 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T00:35:08.946 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package unzip. 2024-04-24T00:35:08.972 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-24T00:35:09.012 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T00:35:09.207 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-24T00:35:09.247 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-24T00:35:09.358 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package luarocks. 2024-04-24T00:35:09.385 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-24T00:35:09.416 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-24T00:35:09.616 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T00:35:09.643 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:09.674 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rbd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librgw2. 2024-04-24T00:35:09.801 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../025-librgw2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:09.849 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librgw2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:09.986 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T00:35:10.012 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-24T00:35:10.052 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-24T00:35:10.401 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T00:35:10.404 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T00:35:10.416 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:10.459 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:10.741 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-24T00:35:10.756 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-24T00:35:10.799 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T00:35:10.812 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-24T00:35:10.839 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T00:35:11.102 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua5.1. 2024-04-24T00:35:11.129 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-24T00:35:11.131 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T00:35:11.158 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:11.160 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-24T00:35:11.189 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libradosstriper1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:11.463 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-any. 2024-04-24T00:35:11.479 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-24T00:35:11.521 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-24T00:35:11.610 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-common. 2024-04-24T00:35:11.636 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:11.774 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package zip. 2024-04-24T00:35:11.790 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-24T00:35:11.832 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking zip (3.0-12build2) ... 2024-04-24T00:35:12.127 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package unzip. 2024-04-24T00:35:12.154 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-24T00:35:12.530 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package luarocks. 2024-04-24T00:35:12.557 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-24T00:35:12.588 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-24T00:35:12.906 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-base. 2024-04-24T00:35:12.933 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:12.963 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librgw2. 2024-04-24T00:35:12.982 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-base (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:12.989 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../025-librgw2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:13.029 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librgw2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:13.486 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-24T00:35:13.512 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-24T00:35:13.544 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-24T00:35:13.573 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T00:35:13.589 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:13.631 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:13.813 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cheroot. 2024-04-24T00:35:13.839 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-24T00:35:13.879 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-24T00:35:14.034 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T00:35:14.191 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-24T00:35:14.218 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-24T00:35:14.258 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-24T00:35:14.337 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T00:35:14.364 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:14.403 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libradosstriper1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:14.577 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-24T00:35:14.603 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-24T00:35:14.643 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-24T00:35:14.959 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-common. 2024-04-24T00:35:14.986 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:15.096 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-24T00:35:15.123 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-24T00:35:15.154 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-24T00:35:15.441 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-tempora. 2024-04-24T00:35:15.468 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-24T00:35:15.507 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-24T00:35:15.810 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-portend. 2024-04-24T00:35:15.837 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-24T00:35:15.876 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-24T00:35:16.171 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-24T00:35:16.198 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-24T00:35:16.237 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-24T00:35:16.364 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-base. 2024-04-24T00:35:16.391 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:16.440 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-base (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:16.541 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T00:35:16.567 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-24T00:35:16.607 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-24T00:35:16.953 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T00:35:16.979 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-24T00:35:17.002 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-24T00:35:17.019 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-24T00:35:17.029 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-24T00:35:17.068 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-24T00:35:17.304 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cheroot. 2024-04-24T00:35:17.320 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-24T00:35:17.330 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T00:35:17.354 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T00:35:17.357 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-24T00:35:17.397 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-24T00:35:17.658 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T00:35:17.665 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-24T00:35:17.685 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-24T00:35:17.692 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-24T00:35:17.716 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-24T00:35:17.732 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-24T00:35:18.002 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-mako. 2024-04-24T00:35:18.029 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-24T00:35:18.035 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-24T00:35:18.062 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-24T00:35:18.069 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T00:35:18.101 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-24T00:35:18.379 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-24T00:35:18.406 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-24T00:35:18.414 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T00:35:18.440 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T00:35:18.446 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-24T00:35:18.480 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T00:35:18.715 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempora. 2024-04-24T00:35:18.742 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-24T00:35:18.773 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-24T00:35:18.783 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T00:35:18.809 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-24T00:35:18.850 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-24T00:35:19.076 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-portend. 2024-04-24T00:35:19.104 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-24T00:35:19.110 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-webob. 2024-04-24T00:35:19.137 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-24T00:35:19.143 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-24T00:35:19.446 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-24T00:35:19.455 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T00:35:19.473 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-24T00:35:19.482 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-24T00:35:19.512 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-24T00:35:19.516 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T00:35:20.111 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T00:35:20.138 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-24T00:35:20.169 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-paste. 2024-04-24T00:35:20.177 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-24T00:35:20.196 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-24T00:35:20.235 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T00:35:20.488 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T00:35:20.516 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-24T00:35:20.530 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T00:35:20.555 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-24T00:35:20.557 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-24T00:35:20.597 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T00:35:20.858 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T00:35:20.885 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-24T00:35:20.891 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T00:35:20.918 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-24T00:35:20.924 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-24T00:35:20.958 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-24T00:35:21.186 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-mako. 2024-04-24T00:35:21.213 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-24T00:35:21.244 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T00:35:21.261 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T00:35:21.288 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-24T00:35:21.327 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-24T00:35:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T00:35:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T00:35:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T00:35:21.605 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T00:35:21.620 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-24T00:35:21.655 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T00:35:21.891 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T00:35:21.918 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-24T00:35:21.958 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-24T00:35:21.977 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-certifi. 2024-04-24T00:35:22.004 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-24T00:35:22.035 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-24T00:35:22.261 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webob. 2024-04-24T00:35:22.288 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-24T00:35:22.327 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-24T00:35:22.371 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-idna. 2024-04-24T00:35:22.398 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-24T00:35:22.437 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-24T00:35:22.689 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T00:35:22.716 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-24T00:35:22.740 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-urllib3. 2024-04-24T00:35:22.767 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-24T00:35:22.777 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T00:35:22.807 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T00:35:23.050 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T00:35:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-24T00:35:23.108 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T00:35:23.127 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-requests. 2024-04-24T00:35:23.153 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-24T00:35:23.193 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T00:35:23.386 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-paste. 2024-04-24T00:35:23.413 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-24T00:35:23.452 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T00:35:23.531 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T00:35:23.747 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T00:35:23.763 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-24T00:35:23.797 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T00:35:23.865 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T00:35:23.892 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:23.932 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:24.075 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-24T00:35:24.312 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T00:35:24.339 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:24.378 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:24.453 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T00:35:24.480 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-24T00:35:24.519 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-24T00:35:24.756 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-24T00:35:24.797 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T00:35:24.813 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-24T00:35:24.847 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T00:35:25.067 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T00:35:25.094 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-24T00:35:25.134 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-24T00:35:25.160 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-certifi. 2024-04-24T00:35:25.177 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-24T00:35:25.428 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T00:35:25.455 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:25.486 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:25.521 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-idna. 2024-04-24T00:35:25.549 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-24T00:35:25.588 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-24T00:35:25.856 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T00:35:25.883 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:25.899 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-urllib3. 2024-04-24T00:35:25.923 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mon (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:25.926 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-24T00:35:25.965 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T00:35:26.310 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests. 2024-04-24T00:35:26.327 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-24T00:35:26.470 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T00:35:26.560 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T00:35:26.586 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:26.626 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-osd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:26.830 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T00:35:26.857 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-24T00:35:26.913 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T00:35:27.175 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T00:35:27.190 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:27.232 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:27.621 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T00:35:27.648 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:27.654 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph. 2024-04-24T00:35:27.669 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../066-ceph_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:27.687 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:27.704 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T00:35:27.967 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-24T00:35:27.990 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T00:35:28.007 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-24T00:35:28.017 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:28.057 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:28.309 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T00:35:28.336 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-24T00:35:28.384 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-24T00:35:28.436 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T00:35:28.461 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mds (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T00:35:28.731 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:28.904 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package cephadm. 2024-04-24T00:35:28.936 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../069-cephadm_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:28.971 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T00:35:29.159 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:29.190 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mon (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:29.240 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-24T00:35:29.267 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-24T00:35:29.298 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-24T00:35:29.609 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-24T00:35:29.627 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T00:35:29.804 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:29.844 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-osd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:30.647 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-babel. 2024-04-24T00:35:30.674 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T00:35:30.714 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T00:35:30.897 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph. 2024-04-24T00:35:30.913 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../066-ceph_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:30.980 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:31.025 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T00:35:31.053 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-24T00:35:31.092 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T00:35:31.283 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T00:35:31.310 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:31.349 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:31.437 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T00:35:31.464 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:31.503 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:31.686 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T00:35:31.713 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:31.752 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mds (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:31.823 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T00:35:31.850 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T00:35:31.889 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T00:35:32.205 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephadm. 2024-04-24T00:35:32.228 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-routes. 2024-04-24T00:35:32.232 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../069-cephadm_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:32.255 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-24T00:35:32.272 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:32.342 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T00:35:32.599 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-24T00:35:32.627 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-24T00:35:32.666 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-24T00:35:32.678 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T00:35:32.706 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:32.745 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:32.960 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-24T00:35:32.977 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T00:35:33.018 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T00:35:33.950 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-24T00:35:33.966 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-24T00:35:34.008 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T00:35:34.074 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-babel. 2024-04-24T00:35:34.101 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T00:35:34.140 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T00:35:34.443 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T00:35:34.470 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-24T00:35:34.509 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T00:35:34.512 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T00:35:34.538 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-24T00:35:34.578 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T00:35:34.854 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T00:35:34.882 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:34.921 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:34.957 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-24T00:35:34.984 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-24T00:35:35.023 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-24T00:35:35.291 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T00:35:35.318 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T00:35:35.326 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T00:35:35.353 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-24T00:35:35.357 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T00:35:35.392 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T00:35:35.643 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-routes. 2024-04-24T00:35:35.670 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-24T00:35:35.709 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T00:35:36.046 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T00:35:36.074 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:36.112 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:36.292 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T00:35:36.320 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:36.444 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:37.242 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-24T00:35:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-24T00:35:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-24T00:35:37.405 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-24T00:35:37.725 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T00:35:37.753 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-24T00:35:37.754 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T00:35:37.782 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-24T00:35:37.791 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-24T00:35:37.821 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T00:35:38.128 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T00:35:38.156 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-24T00:35:38.182 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-24T00:35:38.195 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T00:35:38.210 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-24T00:35:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-24T00:35:38.501 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T00:35:38.518 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-24T00:35:38.523 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-24T00:35:38.868 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T00:35:38.895 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-24T00:35:38.926 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-24T00:35:39.213 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T00:35:39.226 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T00:35:39.240 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-24T00:35:39.255 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:39.279 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T00:35:39.379 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:39.590 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T00:35:39.618 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-24T00:35:39.656 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-24T00:35:39.977 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T00:35:40.004 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-24T00:35:40.046 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T00:35:40.324 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T00:35:40.351 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-24T00:35:40.390 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-24T00:35:40.693 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T00:35:40.698 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T00:35:40.714 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:40.721 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-24T00:35:40.756 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:40.759 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-24T00:35:41.026 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T00:35:41.042 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-24T00:35:41.071 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T00:35:41.076 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T00:35:41.099 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-24T00:35:41.138 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T00:35:41.379 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T00:35:41.406 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T00:35:41.416 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T00:35:41.444 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-24T00:35:41.474 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-24T00:35:41.487 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T00:35:41.740 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package jq. 2024-04-24T00:35:41.756 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T00:35:41.769 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T00:35:41.797 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-24T00:35:41.798 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-24T00:35:41.836 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-24T00:35:42.026 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package socat. 2024-04-24T00:35:42.054 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-24T00:35:42.084 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-24T00:35:42.106 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T00:35:42.133 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-24T00:35:42.164 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T00:35:42.337 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T00:35:42.353 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-24T00:35:42.395 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-24T00:35:42.450 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T00:35:42.478 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-24T00:35:42.516 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-24T00:35:42.791 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-test. 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-test (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T00:35:43.567 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T00:35:43.596 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:43.633 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T00:35:44.356 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T00:35:44.384 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T00:35:44.422 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T00:35:44.616 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package jq. 2024-04-24T00:35:44.633 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T00:35:44.666 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-24T00:35:44.886 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package socat. 2024-04-24T00:35:44.914 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-24T00:35:44.952 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-24T00:35:45.078 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T00:35:45.094 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:45.136 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-volume (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:45.264 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T00:35:45.280 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-24T00:35:45.322 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-24T00:35:45.457 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T00:35:45.473 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:45.515 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libcephfs-dev (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:45.701 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-test. 2024-04-24T00:35:45.718 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:45.759 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-test (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:45.837 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-24T00:35:45.864 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-24T00:35:45.903 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T00:35:46.183 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-24T00:35:46.199 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-24T00:35:46.232 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-24T00:35:46.612 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T00:35:46.640 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-24T00:35:46.678 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T00:35:47.092 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package pkg-config. 2024-04-24T00:35:47.119 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T00:35:47.420 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-24T00:35:47.447 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-24T00:35:47.486 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-24T00:35:47.889 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-24T00:35:47.917 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-24T00:35:47.956 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-24T00:35:48.005 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T00:35:48.033 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:35:48.072 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-volume (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:48.225 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T00:35:48.252 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T00:35:48.283 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:48.450 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs-dev (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:48.553 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T00:35:48.569 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-24T00:35:48.612 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-24T00:35:48.770 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-24T00:35:48.798 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-24T00:35:48.836 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T00:35:48.923 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T00:35:48.950 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-24T00:35:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-24T00:35:49.174 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-24T00:35:49.202 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-24T00:35:49.241 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-24T00:35:49.359 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-py. 2024-04-24T00:35:49.387 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-24T00:35:49.425 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-24T00:35:49.553 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T00:35:49.581 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-24T00:35:49.620 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T00:35:49.754 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pygments. 2024-04-24T00:35:49.781 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-24T00:35:49.820 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T00:35:49.991 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package pkg-config. 2024-04-24T00:35:50.019 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-24T00:35:50.049 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T00:35:50.242 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T00:35:50.745 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-24T00:35:50.745 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-24T00:35:50.745 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-24T00:35:50.745 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-24T00:35:50.746 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-24T00:35:50.746 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-toml. 2024-04-24T00:35:50.746 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-24T00:35:50.746 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-24T00:35:50.988 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T00:35:51.016 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-24T00:35:51.047 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-24T00:35:51.055 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T00:35:51.076 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-24T00:35:51.113 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-24T00:35:51.374 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T00:35:51.391 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T00:35:51.392 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T00:35:51.419 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-24T00:35:51.424 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T00:35:51.458 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-24T00:35:51.703 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T00:35:51.719 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T00:35:51.731 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-24T00:35:51.743 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-24T00:35:51.769 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-24T00:35:52.080 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T00:35:52.108 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-24T00:35:52.147 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-py. 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package radosgw. 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../116-radosgw_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-24T00:35:52.589 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking radosgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:52.886 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pygments. 2024-04-24T00:35:52.914 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-24T00:35:52.952 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T00:35:53.326 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T00:35:53.353 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:53.382 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T00:35:53.392 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking rbd-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:53.410 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-24T00:35:53.448 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-24T00:35:53.679 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package rbd-mirror. 2024-04-24T00:35:53.707 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:53.737 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking rbd-mirror (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:53.751 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-toml. 2024-04-24T00:35:53.779 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-24T00:35:53.817 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-24T00:35:54.120 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T00:35:54.148 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-24T00:35:54.180 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T00:35:54.184 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-24T00:35:54.442 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T00:35:54.517 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T00:35:54.533 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-24T00:35:54.568 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T00:35:54.575 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-24T00:35:54.840 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-24T00:35:54.853 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T00:35:54.881 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-24T00:35:54.919 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-24T00:35:55.166 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-24T00:35:55.587 INFO:teuthology.orchestra.run.smithi134.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-24T00:35:55.662 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:55.723 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package radosgw. 2024-04-24T00:35:55.752 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../116-radosgw_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:55.790 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking radosgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:56.153 INFO:teuthology.orchestra.run.smithi134.stdout:Adding system user cephadm....done 2024-04-24T00:35:56.276 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T00:35:56.618 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:56.651 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking rbd-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:56.829 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-24T00:35:56.955 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-24T00:35:56.995 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package rbd-mirror. 2024-04-24T00:35:57.024 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:35:57.062 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking rbd-mirror (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:57.206 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T00:35:57.474 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T00:35:57.526 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-24T00:35:57.776 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T00:35:57.860 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T00:35:57.885 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T00:35:58.125 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-24T00:35:58.446 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-24T00:35:58.451 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-24T00:35:58.731 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-24T00:35:58.867 INFO:teuthology.orchestra.run.smithi067.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-24T00:35:58.962 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:58.974 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-24T00:35:59.225 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-24T00:35:59.501 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-24T00:35:59.760 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-ceph-argparse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:35:59.862 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user cephadm....done 2024-04-24T00:36:00.010 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T00:36:00.011 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T00:36:00.154 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T00:36:00.270 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-24T00:36:00.281 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T00:36:00.406 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-24T00:36:00.521 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-24T00:36:00.647 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-24T00:36:00.666 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-24T00:36:00.907 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T00:36:01.034 INFO:teuthology.orchestra.run.smithi134.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-24T00:36:01.064 INFO:teuthology.orchestra.run.smithi134.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-24T00:36:01.119 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T00:36:01.237 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-24T00:36:01.239 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T00:36:01.526 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-24T00:36:01.605 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-24T00:36:01.827 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T00:36:01.990 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T00:36:02.187 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-24T00:36:02.242 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-24T00:36:02.447 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-24T00:36:02.518 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up zip (3.0-12build2) ... 2024-04-24T00:36:02.619 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T00:36:02.731 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-24T00:36:03.301 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-24T00:36:03.304 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T00:36:03.572 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T00:36:03.644 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-24T00:36:03.698 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-24T00:36:03.824 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T00:36:03.978 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-argparse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:04.117 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-24T00:36:04.237 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T00:36:04.363 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T00:36:04.490 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T00:36:04.502 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-24T00:36:04.615 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-24T00:36:04.753 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T00:36:04.891 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-24T00:36:05.121 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T00:36:05.279 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-24T00:36:05.314 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-24T00:36:05.395 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T00:36:05.406 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T00:36:05.521 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-24T00:36:05.732 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-24T00:36:05.905 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-24T00:36:06.101 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:06.307 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-idna (3.3-1) ... 2024-04-24T00:36:06.851 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up zip (3.0-12build2) ... 2024-04-24T00:36:06.912 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T00:36:06.952 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T00:36:07.671 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T00:36:07.939 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T00:36:08.191 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T00:36:08.353 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-24T00:36:08.487 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-24T00:36:08.755 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T00:36:08.860 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-24T00:36:08.881 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-24T00:36:08.982 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T00:36:09.102 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T00:36:09.259 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-24T00:36:09.462 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T00:36:09.501 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-24T00:36:09.753 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:36:09.755 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T00:36:09.879 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-24T00:36:10.107 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-24T00:36:10.189 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-24T00:36:10.448 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-24T00:36:10.484 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:10.699 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-24T00:36:10.793 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-idna (3.3-1) ... 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T00:36:11.111 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T00:36:11.177 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-24T00:36:11.436 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-24T00:36:11.838 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T00:36:11.964 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T00:36:12.249 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-24T00:36:12.359 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T00:36:12.533 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-24T00:36:12.593 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:36:12.702 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-24T00:36:12.776 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T00:36:12.902 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-24T00:36:12.954 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T00:36:13.028 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T00:36:13.080 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-24T00:36:13.118 INFO:teuthology.orchestra.run.smithi134.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-24T00:36:13.206 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T00:36:13.364 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T00:36:13.481 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T00:36:13.516 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-24T00:36:13.767 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-24T00:36:13.858 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-24T00:36:14.001 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:36:14.101 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-24T00:36:14.227 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-24T00:36:14.429 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-24T00:36:14.486 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:36:14.612 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T00:36:14.680 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-24T00:36:14.930 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T00:36:14.956 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-24T00:36:15.600 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-24T00:36:15.726 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T00:36:15.835 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-24T00:36:15.868 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-24T00:36:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T00:36:16.220 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T00:36:16.464 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-24T00:36:16.505 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-24T00:36:16.883 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T00:36:17.091 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T00:36:17.159 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T00:36:17.192 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-24T00:36:17.293 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T00:36:17.374 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-24T00:36:17.495 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T00:36:17.619 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T00:36:17.621 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librados2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:17.745 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T00:36:17.747 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:17.873 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-24T00:36:18.147 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-24T00:36:18.201 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libcephfs2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:18.303 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libradosstriper1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:18.415 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-24T00:36:18.420 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-24T00:36:18.541 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-24T00:36:18.697 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librbd1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:18.783 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:36:18.823 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-modules-core (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:18.901 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T00:36:18.924 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:19.117 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T00:36:19.117 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T00:36:19.202 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T00:36:19.584 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libcephfs-dev (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:19.701 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rados (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:19.780 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-24T00:36:19.861 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librgw2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:19.906 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-24T00:36:19.987 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T00:36:20.182 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-24T00:36:20.458 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T00:36:20.795 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rbd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:21.350 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up rbd-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:21.459 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T00:36:21.560 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cephfs (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:21.678 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:21.825 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T00:36:21.927 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librados2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:22.044 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:22.170 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-24T00:36:22.204 INFO:teuthology.orchestra.run.smithi134.stdout:Adding group ceph....done 2024-04-24T00:36:22.514 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:22.547 INFO:teuthology.orchestra.run.smithi134.stdout:Adding system user ceph....done 2024-04-24T00:36:22.640 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libradosstriper1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:22.700 INFO:teuthology.orchestra.run.smithi134.stdout:Setting system user ceph properties....done 2024-04-24T00:36:22.713 INFO:teuthology.orchestra.run.smithi134.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T00:36:22.783 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-24T00:36:22.847 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T00:36:23.051 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librbd1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:23.152 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-modules-core (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:23.167 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T00:36:23.278 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:23.496 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T00:36:23.496 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T00:36:23.655 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-test (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:23.815 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up radosgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:23.963 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs-dev (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:24.089 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rados (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:24.240 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librgw2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:24.289 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T00:36:24.289 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T00:36:24.366 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T00:36:24.758 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-base (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:25.083 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T00:36:25.552 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mds (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rbd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:25.754 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up rbd-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:25.774 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T00:36:25.774 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T00:36:25.880 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:26.006 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cephfs (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:26.140 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up rbd-mirror (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:26.472 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T00:36:26.472 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T00:36:26.649 INFO:teuthology.orchestra.run.smithi067.stdout:Adding group ceph....done 2024-04-24T00:36:26.972 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:27.042 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user ceph....done 2024-04-24T00:36:27.158 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T00:36:27.158 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T00:36:27.194 INFO:teuthology.orchestra.run.smithi067.stdout:Setting system user ceph properties....done 2024-04-24T00:36:27.206 INFO:teuthology.orchestra.run.smithi067.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T00:36:27.357 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T00:36:27.669 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T00:36:27.926 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-osd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:28.125 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-test (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:28.251 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up radosgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:28.262 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T00:36:28.262 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T00:36:28.744 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T00:36:28.744 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T00:36:28.761 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:28.896 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:29.047 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mon (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:29.186 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-base (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:29.260 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T00:36:29.260 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T00:36:29.543 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T00:36:29.723 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:29.858 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:29.984 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-dashboard (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:30.021 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mds (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:30.110 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-volume (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:30.236 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T00:36:30.237 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T00:36:30.455 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-24T00:36:30.564 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-24T00:36:30.714 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up rbd-mirror (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:30.716 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T00:36:31.408 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:31.458 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-24T00:36:31.635 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T00:36:31.635 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T00:36:32.100 INFO:teuthology.orchestra.run.smithi134.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T00:36:32.106 DEBUG:teuthology.parallel:result is None 2024-04-24T00:36:32.125 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-osd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:32.666 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T00:36:32.666 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T00:36:33.138 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:33.281 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:33.433 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mon (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:33.650 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T00:36:33.650 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T00:36:34.101 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:34.243 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:34.352 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-dashboard (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:34.479 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-volume (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:36:34.823 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-24T00:36:34.932 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-24T00:36:35.085 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-24T00:36:36.152 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-24T00:36:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T00:36:36.829 DEBUG:teuthology.parallel:result is None 2024-04-24T00:36:36.829 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:36:36.978 DEBUG:teuthology.orchestra.run.smithi067:> dpkg-query -W -f '${Version}' ceph 2024-04-24T00:36:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:36:36.998 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:36:36.998 INFO:teuthology.task.install:The correct ceph version 19.0.0-2424-gb98aab6c-1jammy is installed. 2024-04-24T00:36:37.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:36:37.140 DEBUG:teuthology.orchestra.run.smithi134:> dpkg-query -W -f '${Version}' ceph 2024-04-24T00:36:37.160 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:36:37.160 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:36:37.160 INFO:teuthology.task.install:The correct ceph version 19.0.0-2424-gb98aab6c-1jammy is installed. 2024-04-24T00:36:37.162 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T00:36:37.162 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:37.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T00:36:37.177 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:37.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T00:36:37.214 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T00:36:37.214 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:37.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T00:36:37.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T00:36:37.287 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:37.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T00:36:37.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T00:36:37.356 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T00:36:37.356 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:37.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T00:36:37.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T00:36:37.423 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:37.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T00:36:37.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T00:36:37.488 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T00:36:37.488 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:37.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T00:36:37.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T00:36:37.557 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:37.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T00:36:37.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T00:36:37.625 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T00:36:37.709 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T00:36:37.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-04-24T00:36:37.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-04-24T00:36:37.724 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T00:36:37.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T00:36:37.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T00:36:37.782 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T00:36:37.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T00:36:37.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T00:36:37.852 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-04-24T00:36:37.852 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T00:36:37.852 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7670535', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9514, 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'sleep_before_teardown': 0, 'subset': '69/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b98aab6c00f483888c049d0d8e7bc507e115dd86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9+zocXEZ6/Q4H0ypnly+tswU5kASmcq3gkzceQm1BjozrYH+u6sbPUXdJsrPoFMUTI0zrUdlvUpdEYYKYMGJE=', 'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK0J1VHusBSk22A7Iy4CLA6n9LTzYYGkZ0qzQ4HEt7zMSj/jgAQFKkF1oxXDprXTf+LUXSk/vsF+zE0l0tfD1W4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'pool', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'journal', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-23_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T00:36:37.853 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-24T00:36:37.881 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-24T00:36:37.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T00:36:37.929 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T00:36:37.942 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:37.943 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T00:36:37.989 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T00:36:37.989 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-04-24T00:36:38.036 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T00:36:38.037 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:36:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-04-24T00:36:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:36:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-24 00:31:00.050006941 +0000 2024-04-24T00:36:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-24 00:30:59.810009084 +0000 2024-04-24T00:36:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-24 00:30:59.810009084 +0000 2024-04-24T00:36:38.037 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-24T00:36:38.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T00:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-24T00:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-24T00:36:38.092 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000429713 s, 1.2 MB/s 2024-04-24T00:36:38.092 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T00:36:38.142 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-04-24T00:36:38.188 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T00:36:38.188 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:36:38.188 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-04-24T00:36:38.188 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:36:38.188 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-24 00:31:00.542002546 +0000 2024-04-24T00:36:38.188 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-24 00:31:00.258005083 +0000 2024-04-24T00:36:38.189 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-24 00:31:00.258005083 +0000 2024-04-24T00:36:38.189 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-24T00:36:38.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T00:36:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-24T00:36:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-24T00:36:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000440393 s, 1.2 MB/s 2024-04-24T00:36:38.245 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T00:36:38.293 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-04-24T00:36:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T00:36:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:36:38.340 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-04-24T00:36:38.340 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:36:38.340 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-24 00:31:01.081997723 +0000 2024-04-24T00:36:38.341 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-24 00:31:00.814000117 +0000 2024-04-24T00:36:38.341 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-24 00:31:00.814000117 +0000 2024-04-24T00:36:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-24T00:36:38.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T00:36:38.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-24T00:36:38.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-24T00:36:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000408393 s, 1.3 MB/s 2024-04-24T00:36:38.396 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T00:36:38.445 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-04-24T00:36:38.492 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T00:36:38.492 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:36:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-04-24T00:36:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:36:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-24 00:31:01.545993578 +0000 2024-04-24T00:36:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-24 00:31:01.293995830 +0000 2024-04-24T00:36:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-24 00:31:01.293995830 +0000 2024-04-24T00:36:38.493 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-24T00:36:38.493 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T00:36:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-24T00:36:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-24T00:36:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000318553 s, 1.6 MB/s 2024-04-24T00:36:38.548 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T00:36:38.597 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T00:36:38.597 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T00:36:38.597 INFO:tasks.ceph:Generating config... 2024-04-24T00:36:38.598 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client] log to stderr = False 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:38.599 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:38.600 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-24T00:36:38.600 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-24T00:36:38.600 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-24T00:36:38.600 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T00:36:38.600 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T00:36:38.600 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T00:36:38.600 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T00:36:38.600 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T00:36:38.600 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-24T00:36:38.600 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-24T00:36:38.600 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-04-24T00:36:38.600 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T00:36:38.600 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T00:36:38.600 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T00:36:38.601 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T00:36:38.601 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T00:36:38.601 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T00:36:38.601 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T00:36:38.601 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T00:36:38.601 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T00:36:38.601 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T00:36:38.601 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T00:36:38.601 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T00:36:38.601 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-04-24T00:36:38.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-04-24T00:36:38.671 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/cluster1.keyring 2024-04-24T00:36:38.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-04-24T00:36:38.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-24T00:36:38.811 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.67')] 2024-04-24T00:36:38.812 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-24T00:36:38.813 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:38.813 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T00:36:38.860 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --print /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T00:36:38.941 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T00:36:38.941 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid 155f5f73-fc57-4025-a4b4-fdcde5f1f25e 2024-04-24T00:36:38.941 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-04-24T00:36:38.941 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-04-24T00:36:38.941 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 155f5f73-fc57-4025-a4b4-fdcde5f1f25e 2024-04-24T00:36:38.941 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-04-24T00:36:38.941592+0000 2024-04-24T00:36:38.942 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-24T00:36:38.941592+0000 2024-04-24T00:36:38.942 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-04-24T00:36:38.942 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-04-24T00:36:38.942 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-04-24T00:36:38.942 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-04-24T00:36:38.943 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T00:36:38.992 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 155f5f73-fc57-4025-a4b4-fdcde5f1f25e... 2024-04-24T00:36:38.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-24T00:36:39.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = 172.21.15.67 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on pool no app = False 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 5000 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 155f5f73-fc57-4025-a4b4-fdcde5f1f25e 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = stupid 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: debug rbd = 20 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: debug rbd_mirror = 15 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: log to stderr = False 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: rbd default features = 125 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: rbd mirror image policy type = simple 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.0] 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.1] 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.2] 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.3] 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = 172.21.15.67 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi134.stdout: mon client directed command retry = 5 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on pool no app = False 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi134.stdout: ms inject socket failures = 5000 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 155f5f73-fc57-4025-a4b4-fdcde5f1f25e 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug trim objects = true 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: bdev async discard = True 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: bdev enable discard = True 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore allocator = stupid 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 1/20 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 1/20 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 4/10 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd = 20 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd_mirror = 15 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: log to stderr = False 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: rbd default features = 125 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: rbd mirror image policy type = simple 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-04-24T00:36:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:39.092 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:39.092 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-04-24T00:36:39.092 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:39.092 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:39.092 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-04-24T00:36:39.092 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:36:39.092 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:36:39.092 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-04-24T00:36:39.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-04-24T00:36:39.170 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T00:36:39.171 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:39.171 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-04-24T00:36:39.177 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:39.178 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-04-24T00:36:39.224 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T00:36:39.224 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:39.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-24T00:36:39.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-24T00:36:39.289 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:39.290 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-24T00:36:39.337 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T00:36:39.337 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:39.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-24T00:36:39.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-24T00:36:39.361 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:39.361 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-24T00:36:39.410 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T00:36:39.411 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T00:36:39.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-24T00:36:39.455 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-24T00:36:39.457 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T00:36:39.458 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T00:36:39.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-04-24T00:36:39.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-04-24T00:36:39.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-24T00:36:39.677 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-24T00:36:39.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-24T00:36:39.759 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-24T00:36:39.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-24T00:36:39.847 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-24T00:36:39.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-24T00:36:39.933 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-24T00:36:39.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-24T00:36:40.014 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-24T00:36:40.024 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-24T00:36:40.096 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-24T00:36:40.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-24T00:36:40.187 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T00:36:40.187 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T00:36:40.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-04-24T00:36:40.201 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T00:36:40.201 INFO:tasks.ceph:role: cluster1.osd.0 2024-04-24T00:36:40.201 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T00:36:40.202 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T00:36:40.264 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:36:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:36:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:36:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:36:40.265 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:36:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:36:40.265 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:36:40.266 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:36:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:36:40.266 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:36:40.725 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-24T00:36:40.727 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-24T00:36:40.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-04-24T00:36:40.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-04-24T00:36:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:36:40.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-04-24T00:36:40.892 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T00:36:40.892 INFO:tasks.ceph:role: cluster1.osd.1 2024-04-24T00:36:40.892 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T00:36:40.892 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T00:36:40.956 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:36:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:36:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:36:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:36:40.956 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:36:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:36:40.957 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:36:40.957 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:36:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:36:40.957 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:36:41.426 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-24T00:36:41.427 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-24T00:36:41.428 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-04-24T00:36:41.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-04-24T00:36:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:36:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:36:41.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-04-24T00:36:41.664 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T00:36:41.664 INFO:tasks.ceph:role: cluster1.osd.2 2024-04-24T00:36:41.664 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T00:36:41.665 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T00:36:41.728 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:36:41.728 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:36:41.728 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:36:41.729 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:36:41.729 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:36:41.729 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:36:41.729 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:36:41.729 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:36:41.729 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:36:41.729 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:36:42.211 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-24T00:36:42.213 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-24T00:36:42.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-04-24T00:36:42.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-04-24T00:36:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:36:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:36:42.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T00:36:42.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:42.475+0000 7fb6654007c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-04-24T00:36:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:42.479+0000 7fb6654007c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-04-24T00:36:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:42.479+0000 7fb6654007c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-24T00:36:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:42.479+0000 7fb6654007c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-24T00:36:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:42.479+0000 7fb6654007c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-24T00:36:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:42.479+0000 7fb6654007c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-04-24T00:36:45.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-04-24T00:36:45.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T00:36:45.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:45.571+0000 7f3a933817c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-04-24T00:36:45.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:45.571+0000 7f3a933817c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-04-24T00:36:45.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:45.571+0000 7f3a933817c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-24T00:36:45.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:45.571+0000 7f3a933817c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-24T00:36:45.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:45.571+0000 7f3a933817c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-24T00:36:45.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:45.571+0000 7f3a933817c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-04-24T00:36:48.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-04-24T00:36:48.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T00:36:48.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:48.687+0000 7f94d5b3f7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-04-24T00:36:48.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:48.687+0000 7f94d5b3f7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-04-24T00:36:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:48.687+0000 7f94d5b3f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-24T00:36:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:48.687+0000 7f94d5b3f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-24T00:36:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:48.687+0000 7f94d5b3f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-24T00:36:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-24T00:36:48.687+0000 7f94d5b3f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-04-24T00:36:51.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-04-24T00:36:51.725 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T00:36:51.725 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:51.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-04-24T00:36:51.780 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:51.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-04-24T00:36:51.835 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:51.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-04-24T00:36:51.891 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:51.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-04-24T00:36:51.953 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:51.954 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-04-24T00:36:52.000 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:52.000 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-04-24T00:36:52.006 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:52.007 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-04-24T00:36:52.053 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:52.054 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-04-24T00:36:52.101 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:52.102 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-04-24T00:36:52.149 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:52.150 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-04-24T00:36:52.198 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:52.198 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-04-24T00:36:52.245 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:52.246 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-04-24T00:36:52.293 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:36:52.294 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-04-24T00:36:52.341 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T00:36:52.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/cluster1.keyring 2024-04-24T00:36:52.354 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-04-24T00:36:52.354 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAXVChmRlEiGxAAawy1+FVgbnLNFbEtE4HIeQ== 2024-04-24T00:36:52.354 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-04-24T00:36:52.354 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAaVChmA6SZHBAAfxFfQ3ygHaRosaFXkDWszA== 2024-04-24T00:36:52.354 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-04-24T00:36:52.354 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAdVChmqeggIhAAfT8mw24P1aXJvWQ6bWaiwg== 2024-04-24T00:36:52.354 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-04-24T00:36:52.354 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAgVChmilMFKRAARFaaFjAbsar9618D/WRG4Q== 2024-04-24T00:36:52.354 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAXVChmXDy8HxAAw4yAEOnfRNJRSwVMY+6b/Q== 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror] 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAXVChmRKiIIhAAldBUuPvYcu9e1TP8PhOoog== 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.0] 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAXVChmgZhlKBAAbeHibthsjhfWhU1qhLcAqA== 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.1] 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAXVChmWKpILRAACSglusKKt6eWUdY5A96mfA== 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.2] 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAXVChmRPqKMhAAb7x2QliTCYuJm6zDkDnxfw== 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.3] 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAXVChmYQunNxAAYAzk15KY1yveX64xvruefw== 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.4] 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAYVChmnzbhABAAhdDTFYL1mnY7HjRzcXP4eA== 2024-04-24T00:36:52.355 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.5] 2024-04-24T00:36:52.356 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAYVChmWXG4BRAApdFxV9RWHlKIfk2Bmt3RuA== 2024-04-24T00:36:52.356 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.6] 2024-04-24T00:36:52.356 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAYVChm2eqdChAAoZqxvpcL5034pXpUi5l9+g== 2024-04-24T00:36:52.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T00:36:52.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:36:52.492 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:36:52.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:36:52.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:36:52.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:36:52.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:36:52.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:36:53.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:36:53.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:36:53.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:36:53.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:36:53.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:36:53.368 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T00:36:53.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-04-24T00:36:53.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-04-24T00:36:53.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-04-24T00:36:53.523 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T00:36:53.570 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-04-24T00:36:53.570 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-04-24T00:36:53.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-04-24T00:36:53.613 INFO:tasks.ceph.cluster1.mon.a:Started 2024-04-24T00:36:53.613 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-04-24T00:36:53.613 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-04-24T00:36:53.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-04-24T00:36:53.616 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-04-24T00:36:53.616 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T00:36:53.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd crush tunables default 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-04-24T00:36:53.879 INFO:tasks.ceph:check_enable_crimson: False 2024-04-24T00:36:53.879 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-04-24T00:36:53.879 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:53.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-04-24T00:36:53.890 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:53.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-04-24T00:36:53.941 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:36:53.941 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-04-24T00:36:53.944 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:53.943+0000 7fd266652180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T00:36:53.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd new 2368b4cd-28d5-4a67-906e-4559d1b49adb 0 2024-04-24T00:36:54.194 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-04-24T00:36:54.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd new 544e79cb-b3f4-4a0a-b050-10129b035da2 1 2024-04-24T00:36:54.408 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-24T00:36:54.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd new 90e497de-f375-4463-be63-b7dadec011e4 2 2024-04-24T00:36:54.427 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:54.423+0000 7fd266652180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T00:36:54.632 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:54.631+0000 7fd266652180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi067.stdout:2 2024-04-24T00:36:54.660 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-04-24T00:36:54.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-04-24T00:36:54.662 INFO:tasks.ceph.cluster1.osd.0:Started 2024-04-24T00:36:54.663 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-04-24T00:36:54.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-04-24T00:36:54.664 INFO:tasks.ceph.cluster1.osd.1:Started 2024-04-24T00:36:54.665 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-04-24T00:36:54.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-04-24T00:36:54.667 INFO:tasks.ceph.cluster1.osd.2:Started 2024-04-24T00:36:54.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T00:36:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:36:54.847 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"fsid":"155f5f73-fc57-4025-a4b4-fdcde5f1f25e","created":"2024-04-24T00:36:53.725251+0000","modified":"2024-04-24T00:36:54.643561+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T00:36:54.643561+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2368b4cd-28d5-4a67-906e-4559d1b49adb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"544e79cb-b3f4-4a0a-b050-10129b035da2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90e497de-f375-4463-be63-b7dadec011e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:36:54.862 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-04-24T00:36:54.863 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T00:36:55.165 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-24T00:36:55.228 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:55.227+0000 7fd266652180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T00:36:55.285 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:55.283+0000 7fd266652180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T00:36:55.288 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:36:55.287+0000 7fbc4e4807c0 -1 Falling back to public interface 2024-04-24T00:36:55.300 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:36:55.299+0000 7efd71f297c0 -1 Falling back to public interface 2024-04-24T00:36:55.301 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:36:55.299+0000 7fd9631007c0 -1 Falling back to public interface 2024-04-24T00:36:55.343 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:55.339+0000 7fd266652180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T00:36:55.351 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-24T00:36:55.351 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":5,"fsid":"155f5f73-fc57-4025-a4b4-fdcde5f1f25e","created":"2024-04-24T00:36:53.725251+0000","modified":"2024-04-24T00:36:54.643561+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T00:36:54.643561+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2368b4cd-28d5-4a67-906e-4559d1b49adb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"544e79cb-b3f4-4a0a-b050-10129b035da2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90e497de-f375-4463-be63-b7dadec011e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:36:55.366 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T00:36:55.425 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:55.423+0000 7fd266652180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T00:36:55.480 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:55.479+0000 7fd266652180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T00:36:55.560 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:55.559+0000 7fd266652180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T00:36:55.615 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:55.615+0000 7fd266652180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T00:36:55.773 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:55.771+0000 7fd266652180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T00:36:55.838 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:55.835+0000 7fd266652180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T00:36:56.526 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-24T00:36:56.526 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-24T00:36:56.526 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr: from numpy import show_config as show_numpy_config 2024-04-24T00:36:56.615 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:56.611+0000 7fd266652180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T00:36:56.808 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:56.807+0000 7fd266652180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T00:36:57.000 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:56.999+0000 7fd266652180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T00:36:57.071 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:57.067+0000 7fd266652180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T00:36:57.222 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:57.219+0000 7fd266652180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T00:36:57.872 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:57.871+0000 7fd266652180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T00:36:57.950 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:57.947+0000 7fd266652180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T00:36:58.140 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:58.139+0000 7fd266652180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T00:36:58.229 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:58.227+0000 7fd266652180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T00:36:58.557 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:58.555+0000 7fd266652180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T00:36:58.622 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:58.619+0000 7fd266652180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T00:36:58.939 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:36:58.935+0000 7efd71f297c0 -1 osd.0 0 log_to_monitors true 2024-04-24T00:36:58.961 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:36:58.959+0000 7fbc4e4807c0 -1 osd.2 0 log_to_monitors true 2024-04-24T00:36:58.990 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:36:58.987+0000 7fd9631007c0 -1 osd.1 0 log_to_monitors true 2024-04-24T00:36:59.656 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-04-24T00:36:59.655+0000 7fd20da99640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-24T00:37:01.669 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-24T00:37:02.089 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-24T00:37:02.089 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":7,"fsid":"155f5f73-fc57-4025-a4b4-fdcde5f1f25e","created":"2024-04-24T00:36:53.725251+0000","modified":"2024-04-24T00:37:00.699571+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T00:36:54.643561+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2368b4cd-28d5-4a67-906e-4559d1b49adb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"544e79cb-b3f4-4a0a-b050-10129b035da2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90e497de-f375-4463-be63-b7dadec011e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:02.114 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T00:37:02.699 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:37:02.695+0000 7efd6d6af640 -1 osd.0 0 waiting for initial osdmap 2024-04-24T00:37:02.718 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:37:02.715+0000 7fbc49c06640 -1 osd.2 0 waiting for initial osdmap 2024-04-24T00:37:02.728 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:37:02.727+0000 7efd65cf6640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:37:02.748 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:37:02.747+0000 7fbc4224d640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:37:02.763 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:37:02.763+0000 7fd9600bc640 -1 osd.1 0 waiting for initial osdmap 2024-04-24T00:37:02.796 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:37:02.795+0000 7fd956ecd640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:37:08.418 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-24T00:37:08.722 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-24T00:37:08.722 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"155f5f73-fc57-4025-a4b4-fdcde5f1f25e","created":"2024-04-24T00:36:53.725251+0000","modified":"2024-04-24T00:37:06.756311+0000","last_up_change":"2024-04-24T00:37:03.735676+0000","last_in_change":"2024-04-24T00:36:54.643561+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:37:04.722047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2368b4cd-28d5-4a67-906e-4559d1b49adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6810","nonce":290552072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6814","nonce":290552072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6821","nonce":290552072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6817","nonce":290552072}]},"public_addr":"172.21.15.67:6810/290552072","cluster_addr":"172.21.15.67:6814/290552072","heartbeat_back_addr":"172.21.15.67:6821/290552072","heartbeat_front_addr":"172.21.15.67:6817/290552072","state":["exists","up"]},{"osd":1,"uuid":"544e79cb-b3f4-4a0a-b050-10129b035da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6811","nonce":86988388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6816","nonce":86988388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6823","nonce":86988388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6820","nonce":86988388}]},"public_addr":"172.21.15.67:6811/86988388","cluster_addr":"172.21.15.67:6816/86988388","heartbeat_back_addr":"172.21.15.67:6823/86988388","heartbeat_front_addr":"172.21.15.67:6820/86988388","state":["exists","up"]},{"osd":2,"uuid":"90e497de-f375-4463-be63-b7dadec011e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6801","nonce":1778384542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6803","nonce":1778384542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6807","nonce":1778384542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6805","nonce":1778384542}]},"public_addr":"172.21.15.67:6801/1778384542","cluster_addr":"172.21.15.67:6803/1778384542","heartbeat_back_addr":"172.21.15.67:6807/1778384542","heartbeat_front_addr":"172.21.15.67:6805/1778384542","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:36:59.908287+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:00.010705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:36:59.930058+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:08.740 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-24T00:37:08.740 INFO:tasks.ceph:Creating RBD pool 2024-04-24T00:37:08.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-04-24T00:37:09.773 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'rbd' created 2024-04-24T00:37:09.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T00:37:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T00:37:10.807 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-04-24T00:37:10.808 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-04-24T00:37:10.808 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-04-24T00:37:10.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-04-24T00:37:10.808 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T00:37:11.153 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:11.170 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3045299875},{"type":"v1","addr":"172.21.15.67:6825","nonce":3045299875}]},"active_addr":"172.21.15.67:6825/3045299875","active_change":"2024-04-24T00:36:58.640999+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2059159943}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1491314016}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2820601342}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1311356233}]}]} 2024-04-24T00:37:11.175 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-04-24T00:37:11.175 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-04-24T00:37:11.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T00:37:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:11.470 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":13,"fsid":"155f5f73-fc57-4025-a4b4-fdcde5f1f25e","created":"2024-04-24T00:36:53.725251+0000","modified":"2024-04-24T00:37:10.773520+0000","last_up_change":"2024-04-24T00:37:03.735676+0000","last_in_change":"2024-04-24T00:36:54.643561+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:37:04.722047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:37:09.074949+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2368b4cd-28d5-4a67-906e-4559d1b49adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6810","nonce":290552072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6814","nonce":290552072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6821","nonce":290552072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6817","nonce":290552072}]},"public_addr":"172.21.15.67:6810/290552072","cluster_addr":"172.21.15.67:6814/290552072","heartbeat_back_addr":"172.21.15.67:6821/290552072","heartbeat_front_addr":"172.21.15.67:6817/290552072","state":["exists","up"]},{"osd":1,"uuid":"544e79cb-b3f4-4a0a-b050-10129b035da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6811","nonce":86988388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6816","nonce":86988388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6823","nonce":86988388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6820","nonce":86988388}]},"public_addr":"172.21.15.67:6811/86988388","cluster_addr":"172.21.15.67:6816/86988388","heartbeat_back_addr":"172.21.15.67:6823/86988388","heartbeat_front_addr":"172.21.15.67:6820/86988388","state":["exists","up"]},{"osd":2,"uuid":"90e497de-f375-4463-be63-b7dadec011e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6801","nonce":1778384542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6803","nonce":1778384542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6807","nonce":1778384542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6805","nonce":1778384542}]},"public_addr":"172.21.15.67:6801/1778384542","cluster_addr":"172.21.15.67:6803/1778384542","heartbeat_back_addr":"172.21.15.67:6807/1778384542","heartbeat_front_addr":"172.21.15.67:6805/1778384542","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:36:59.908287+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:00.010705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:36:59.930058+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:11.489 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-04-24T00:37:11.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T00:37:11.787 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:11.787 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":14,"fsid":"155f5f73-fc57-4025-a4b4-fdcde5f1f25e","created":"2024-04-24T00:36:53.725251+0000","modified":"2024-04-24T00:37:11.780004+0000","last_up_change":"2024-04-24T00:37:03.735676+0000","last_in_change":"2024-04-24T00:36:54.643561+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:37:04.722047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:37:09.074949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2368b4cd-28d5-4a67-906e-4559d1b49adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6810","nonce":290552072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6814","nonce":290552072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6821","nonce":290552072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6817","nonce":290552072}]},"public_addr":"172.21.15.67:6810/290552072","cluster_addr":"172.21.15.67:6814/290552072","heartbeat_back_addr":"172.21.15.67:6821/290552072","heartbeat_front_addr":"172.21.15.67:6817/290552072","state":["exists","up"]},{"osd":1,"uuid":"544e79cb-b3f4-4a0a-b050-10129b035da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6811","nonce":86988388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6816","nonce":86988388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6823","nonce":86988388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6820","nonce":86988388}]},"public_addr":"172.21.15.67:6811/86988388","cluster_addr":"172.21.15.67:6816/86988388","heartbeat_back_addr":"172.21.15.67:6823/86988388","heartbeat_front_addr":"172.21.15.67:6820/86988388","state":["exists","up"]},{"osd":2,"uuid":"90e497de-f375-4463-be63-b7dadec011e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6801","nonce":1778384542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6803","nonce":1778384542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6807","nonce":1778384542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6805","nonce":1778384542}]},"public_addr":"172.21.15.67:6801/1778384542","cluster_addr":"172.21.15.67:6803/1778384542","heartbeat_back_addr":"172.21.15.67:6807/1778384542","heartbeat_front_addr":"172.21.15.67:6805/1778384542","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:36:59.908287+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:00.010705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:36:59.930058+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:11.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-04-24T00:37:11.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-04-24T00:37:11.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-04-24T00:37:11.964 INFO:teuthology.orchestra.run.smithi067.stdout:34359738374 2024-04-24T00:37:11.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-24T00:37:11.984 INFO:teuthology.orchestra.run.smithi067.stdout:34359738375 2024-04-24T00:37:11.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-24T00:37:11.985 INFO:teuthology.orchestra.run.smithi067.stdout:34359738374 2024-04-24T00:37:11.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi067.stdout:34359738373 2024-04-24T00:37:12.267 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.1 2024-04-24T00:37:12.309 INFO:teuthology.orchestra.run.smithi067.stdout:34359738373 2024-04-24T00:37:12.321 INFO:teuthology.orchestra.run.smithi067.stdout:34359738374 2024-04-24T00:37:12.326 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.2 2024-04-24T00:37:12.338 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738374 for osd.0 2024-04-24T00:37:13.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-24T00:37:13.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-24T00:37:13.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-24T00:37:13.581 INFO:teuthology.orchestra.run.smithi067.stdout:34359738374 2024-04-24T00:37:13.598 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.1 2024-04-24T00:37:13.599 DEBUG:teuthology.parallel:result is None 2024-04-24T00:37:13.618 INFO:teuthology.orchestra.run.smithi067.stdout:34359738374 2024-04-24T00:37:13.635 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2024-04-24T00:37:13.635 DEBUG:teuthology.parallel:result is None 2024-04-24T00:37:13.666 INFO:teuthology.orchestra.run.smithi067.stdout:34359738375 2024-04-24T00:37:13.682 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738375 for osd.0 2024-04-24T00:37:13.682 DEBUG:teuthology.parallel:result is None 2024-04-24T00:37:13.682 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-24T00:37:13.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T00:37:14.001 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:14.002 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-24T00:37:14.018 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-24T00:37:12.650501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82572,"kb_used_data":1504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032948,"statfs":{"total":289910292480,"available":289825738752,"internally_reserved":0,"allocated":1540096,"data_stored":1278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.875930"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790626+0000","last_change":"2024-04-24T00:37:10.788095+0000","last_active":"2024-04-24T00:37:10.790626+0000","last_peered":"2024-04-24T00:37:10.790626+0000","last_clean":"2024-04-24T00:37:10.790626+0000","last_became_active":"2024-04-24T00:37:10.787769+0000","last_became_peered":"2024-04-24T00:37:10.787769+0000","last_unstale":"2024-04-24T00:37:10.790626+0000","last_undegraded":"2024-04-24T00:37:10.790626+0000","last_fullsized":"2024-04-24T00:37:10.790626+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:38.191477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790615+0000","last_change":"2024-04-24T00:37:10.788073+0000","last_active":"2024-04-24T00:37:10.790615+0000","last_peered":"2024-04-24T00:37:10.790615+0000","last_clean":"2024-04-24T00:37:10.790615+0000","last_became_active":"2024-04-24T00:37:10.787658+0000","last_became_peered":"2024-04-24T00:37:10.787658+0000","last_unstale":"2024-04-24T00:37:10.790615+0000","last_undegraded":"2024-04-24T00:37:10.790615+0000","last_fullsized":"2024-04-24T00:37:10.790615+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:20:21.176952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790354+0000","last_change":"2024-04-24T00:37:10.785452+0000","last_active":"2024-04-24T00:37:10.790354+0000","last_peered":"2024-04-24T00:37:10.790354+0000","last_clean":"2024-04-24T00:37:10.790354+0000","last_became_active":"2024-04-24T00:37:10.785100+0000","last_became_peered":"2024-04-24T00:37:10.785100+0000","last_unstale":"2024-04-24T00:37:10.790354+0000","last_undegraded":"2024-04-24T00:37:10.790354+0000","last_fullsized":"2024-04-24T00:37:10.790354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:44:17.001874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790614+0000","last_change":"2024-04-24T00:37:10.787976+0000","last_active":"2024-04-24T00:37:10.790614+0000","last_peered":"2024-04-24T00:37:10.790614+0000","last_clean":"2024-04-24T00:37:10.790614+0000","last_became_active":"2024-04-24T00:37:10.787561+0000","last_became_peered":"2024-04-24T00:37:10.787561+0000","last_unstale":"2024-04-24T00:37:10.790614+0000","last_undegraded":"2024-04-24T00:37:10.790614+0000","last_fullsized":"2024-04-24T00:37:10.790614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:27.735073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T00:37:11.784913+0000","last_change":"2024-04-24T00:37:10.786163+0000","last_active":"2024-04-24T00:37:11.784913+0000","last_peered":"2024-04-24T00:37:11.784913+0000","last_clean":"2024-04-24T00:37:11.784913+0000","last_became_active":"2024-04-24T00:37:10.785718+0000","last_became_peered":"2024-04-24T00:37:10.785718+0000","last_unstale":"2024-04-24T00:37:11.784913+0000","last_undegraded":"2024-04-24T00:37:11.784913+0000","last_fullsized":"2024-04-24T00:37:11.784913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:57.680592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.789218+0000","last_change":"2024-04-24T00:37:10.786864+0000","last_active":"2024-04-24T00:37:10.789218+0000","last_peered":"2024-04-24T00:37:10.789218+0000","last_clean":"2024-04-24T00:37:10.789218+0000","last_became_active":"2024-04-24T00:37:10.785638+0000","last_became_peered":"2024-04-24T00:37:10.785638+0000","last_unstale":"2024-04-24T00:37:10.789218+0000","last_undegraded":"2024-04-24T00:37:10.789218+0000","last_fullsized":"2024-04-24T00:37:10.789218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:46.193184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.789175+0000","last_change":"2024-04-24T00:37:10.786316+0000","last_active":"2024-04-24T00:37:10.789175+0000","last_peered":"2024-04-24T00:37:10.789175+0000","last_clean":"2024-04-24T00:37:10.789175+0000","last_became_active":"2024-04-24T00:37:10.785901+0000","last_became_peered":"2024-04-24T00:37:10.785901+0000","last_unstale":"2024-04-24T00:37:10.789175+0000","last_undegraded":"2024-04-24T00:37:10.789175+0000","last_fullsized":"2024-04-24T00:37:10.789175+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:19:44.178239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790367+0000","last_change":"2024-04-24T00:37:10.785726+0000","last_active":"2024-04-24T00:37:10.790367+0000","last_peered":"2024-04-24T00:37:10.790367+0000","last_clean":"2024-04-24T00:37:10.790367+0000","last_became_active":"2024-04-24T00:37:10.785333+0000","last_became_peered":"2024-04-24T00:37:10.785333+0000","last_unstale":"2024-04-24T00:37:10.790367+0000","last_undegraded":"2024-04-24T00:37:10.790367+0000","last_fullsized":"2024-04-24T00:37:10.790367+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:07:43.616865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":143,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.782753+0000","last_change":"2024-04-24T00:37:05.801930+0000","last_active":"2024-04-24T00:37:10.782753+0000","last_peered":"2024-04-24T00:37:10.782753+0000","last_clean":"2024-04-24T00:37:10.782753+0000","last_became_active":"2024-04-24T00:37:05.799306+0000","last_became_peered":"2024-04-24T00:37:05.799306+0000","last_unstale":"2024-04-24T00:37:10.782753+0000","last_undegraded":"2024-04-24T00:37:10.782753+0000","last_fullsized":"2024-04-24T00:37:10.782753+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_clean_scrub_stamp":"2024-04-24T00:37:04.744801+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:37:53.503175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:37:14.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T00:37:14.310 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:14.311 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-24T00:37:14.327 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-24T00:37:12.650501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82572,"kb_used_data":1504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032948,"statfs":{"total":289910292480,"available":289825738752,"internally_reserved":0,"allocated":1540096,"data_stored":1278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.875930"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790626+0000","last_change":"2024-04-24T00:37:10.788095+0000","last_active":"2024-04-24T00:37:10.790626+0000","last_peered":"2024-04-24T00:37:10.790626+0000","last_clean":"2024-04-24T00:37:10.790626+0000","last_became_active":"2024-04-24T00:37:10.787769+0000","last_became_peered":"2024-04-24T00:37:10.787769+0000","last_unstale":"2024-04-24T00:37:10.790626+0000","last_undegraded":"2024-04-24T00:37:10.790626+0000","last_fullsized":"2024-04-24T00:37:10.790626+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:38.191477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790615+0000","last_change":"2024-04-24T00:37:10.788073+0000","last_active":"2024-04-24T00:37:10.790615+0000","last_peered":"2024-04-24T00:37:10.790615+0000","last_clean":"2024-04-24T00:37:10.790615+0000","last_became_active":"2024-04-24T00:37:10.787658+0000","last_became_peered":"2024-04-24T00:37:10.787658+0000","last_unstale":"2024-04-24T00:37:10.790615+0000","last_undegraded":"2024-04-24T00:37:10.790615+0000","last_fullsized":"2024-04-24T00:37:10.790615+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:20:21.176952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790354+0000","last_change":"2024-04-24T00:37:10.785452+0000","last_active":"2024-04-24T00:37:10.790354+0000","last_peered":"2024-04-24T00:37:10.790354+0000","last_clean":"2024-04-24T00:37:10.790354+0000","last_became_active":"2024-04-24T00:37:10.785100+0000","last_became_peered":"2024-04-24T00:37:10.785100+0000","last_unstale":"2024-04-24T00:37:10.790354+0000","last_undegraded":"2024-04-24T00:37:10.790354+0000","last_fullsized":"2024-04-24T00:37:10.790354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:44:17.001874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790614+0000","last_change":"2024-04-24T00:37:10.787976+0000","last_active":"2024-04-24T00:37:10.790614+0000","last_peered":"2024-04-24T00:37:10.790614+0000","last_clean":"2024-04-24T00:37:10.790614+0000","last_became_active":"2024-04-24T00:37:10.787561+0000","last_became_peered":"2024-04-24T00:37:10.787561+0000","last_unstale":"2024-04-24T00:37:10.790614+0000","last_undegraded":"2024-04-24T00:37:10.790614+0000","last_fullsized":"2024-04-24T00:37:10.790614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:27.735073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T00:37:11.784913+0000","last_change":"2024-04-24T00:37:10.786163+0000","last_active":"2024-04-24T00:37:11.784913+0000","last_peered":"2024-04-24T00:37:11.784913+0000","last_clean":"2024-04-24T00:37:11.784913+0000","last_became_active":"2024-04-24T00:37:10.785718+0000","last_became_peered":"2024-04-24T00:37:10.785718+0000","last_unstale":"2024-04-24T00:37:11.784913+0000","last_undegraded":"2024-04-24T00:37:11.784913+0000","last_fullsized":"2024-04-24T00:37:11.784913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:57.680592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.789218+0000","last_change":"2024-04-24T00:37:10.786864+0000","last_active":"2024-04-24T00:37:10.789218+0000","last_peered":"2024-04-24T00:37:10.789218+0000","last_clean":"2024-04-24T00:37:10.789218+0000","last_became_active":"2024-04-24T00:37:10.785638+0000","last_became_peered":"2024-04-24T00:37:10.785638+0000","last_unstale":"2024-04-24T00:37:10.789218+0000","last_undegraded":"2024-04-24T00:37:10.789218+0000","last_fullsized":"2024-04-24T00:37:10.789218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:46.193184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.789175+0000","last_change":"2024-04-24T00:37:10.786316+0000","last_active":"2024-04-24T00:37:10.789175+0000","last_peered":"2024-04-24T00:37:10.789175+0000","last_clean":"2024-04-24T00:37:10.789175+0000","last_became_active":"2024-04-24T00:37:10.785901+0000","last_became_peered":"2024-04-24T00:37:10.785901+0000","last_unstale":"2024-04-24T00:37:10.789175+0000","last_undegraded":"2024-04-24T00:37:10.789175+0000","last_fullsized":"2024-04-24T00:37:10.789175+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:19:44.178239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.790367+0000","last_change":"2024-04-24T00:37:10.785726+0000","last_active":"2024-04-24T00:37:10.790367+0000","last_peered":"2024-04-24T00:37:10.790367+0000","last_clean":"2024-04-24T00:37:10.790367+0000","last_became_active":"2024-04-24T00:37:10.785333+0000","last_became_peered":"2024-04-24T00:37:10.785333+0000","last_unstale":"2024-04-24T00:37:10.790367+0000","last_undegraded":"2024-04-24T00:37:10.790367+0000","last_fullsized":"2024-04-24T00:37:10.790367+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:07:43.616865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":143,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T00:37:10.782753+0000","last_change":"2024-04-24T00:37:05.801930+0000","last_active":"2024-04-24T00:37:10.782753+0000","last_peered":"2024-04-24T00:37:10.782753+0000","last_clean":"2024-04-24T00:37:10.782753+0000","last_became_active":"2024-04-24T00:37:05.799306+0000","last_became_peered":"2024-04-24T00:37:05.799306+0000","last_unstale":"2024-04-24T00:37:10.782753+0000","last_undegraded":"2024-04-24T00:37:10.782753+0000","last_fullsized":"2024-04-24T00:37:10.782753+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_clean_scrub_stamp":"2024-04-24T00:37:04.744801+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:37:53.503175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:37:14.328 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-24T00:37:14.328 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-04-24T00:37:14.328 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-04-24T00:37:14.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-04-24T00:37:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:14.650 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T00:37:14.667 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-04-24T00:37:14.668 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T00:37:14.678 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-04-24T00:37:14.678 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T00:37:14.678 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7670535', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9514, 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'sleep_before_teardown': 0, 'subset': '69/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b98aab6c00f483888c049d0d8e7bc507e115dd86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9+zocXEZ6/Q4H0ypnly+tswU5kASmcq3gkzceQm1BjozrYH+u6sbPUXdJsrPoFMUTI0zrUdlvUpdEYYKYMGJE=', 'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK0J1VHusBSk22A7Iy4CLA6n9LTzYYGkZ0qzQ4HEt7zMSj/jgAQFKkF1oxXDprXTf+LUXSk/vsF+zE0l0tfD1W4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'pool', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'journal', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-23_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T00:37:14.679 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-24T00:37:14.680 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-24T00:37:14.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T00:37:14.728 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T00:37:14.742 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:14.742 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T00:37:14.789 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T00:37:14.789 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-04-24T00:37:14.837 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T00:37:14.837 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:37:14.837 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-04-24T00:37:14.838 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:37:14.838 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-24 00:31:00.031711625 +0000 2024-04-24T00:37:14.838 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-24 00:30:59.767713980 +0000 2024-04-24T00:37:14.838 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-24 00:30:59.767713980 +0000 2024-04-24T00:37:14.838 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2024-04-24T00:37:14.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T00:37:14.891 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-24T00:37:14.891 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-24T00:37:14.891 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000367321 s, 1.4 MB/s 2024-04-24T00:37:14.893 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T00:37:14.943 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-04-24T00:37:14.989 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T00:37:14.989 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:37:14.989 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-04-24T00:37:14.989 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:37:14.989 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-24 00:31:00.543707057 +0000 2024-04-24T00:37:14.989 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-24 00:31:00.287709341 +0000 2024-04-24T00:37:14.990 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-24 00:31:00.287709341 +0000 2024-04-24T00:37:14.990 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2024-04-24T00:37:14.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T00:37:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-24T00:37:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-24T00:37:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000342756 s, 1.5 MB/s 2024-04-24T00:37:15.044 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T00:37:15.090 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-04-24T00:37:15.138 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T00:37:15.138 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:37:15.138 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-04-24T00:37:15.138 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:37:15.139 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-24 00:31:01.031702703 +0000 2024-04-24T00:37:15.139 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-24 00:31:00.775704988 +0000 2024-04-24T00:37:15.139 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-24 00:31:00.775704988 +0000 2024-04-24T00:37:15.139 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2024-04-24T00:37:15.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T00:37:15.191 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-24T00:37:15.191 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-24T00:37:15.191 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000413926 s, 1.2 MB/s 2024-04-24T00:37:15.192 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T00:37:15.244 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-04-24T00:37:15.293 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T00:37:15.293 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:37:15.293 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-04-24T00:37:15.293 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:37:15.293 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-24 00:31:01.515698387 +0000 2024-04-24T00:37:15.294 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-24 00:31:01.251700741 +0000 2024-04-24T00:37:15.294 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-24 00:31:01.251700741 +0000 2024-04-24T00:37:15.294 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2024-04-24T00:37:15.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T00:37:15.347 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-24T00:37:15.347 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-24T00:37:15.348 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00035761 s, 1.4 MB/s 2024-04-24T00:37:15.349 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T00:37:15.400 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T00:37:15.400 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T00:37:15.400 INFO:tasks.ceph:Generating config... 2024-04-24T00:37:15.401 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-24T00:37:15.401 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-24T00:37:15.401 INFO:tasks.ceph:[client] log to stderr = False 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.402 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.402 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-24T00:37:15.402 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-24T00:37:15.402 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-24T00:37:15.403 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T00:37:15.403 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T00:37:15.403 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T00:37:15.403 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T00:37:15.403 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T00:37:15.403 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T00:37:15.404 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T00:37:15.404 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T00:37:15.404 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T00:37:15.404 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T00:37:15.404 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-04-24T00:37:15.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-04-24T00:37:15.477 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.keyring 2024-04-24T00:37:15.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-04-24T00:37:15.554 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-24T00:37:15.608 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.134')] 2024-04-24T00:37:15.608 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.134', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-24T00:37:15.609 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:15.609 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T00:37:15.653 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.134 --print /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T00:37:15.732 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T00:37:15.732 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: generated fsid f9adc1f2-0792-4cd2-afe7-fbec5c431238 2024-04-24T00:37:15.732 INFO:teuthology.orchestra.run.smithi134.stdout:setting min_mon_release = quincy 2024-04-24T00:37:15.733 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 0 2024-04-24T00:37:15.733 INFO:teuthology.orchestra.run.smithi134.stdout:fsid f9adc1f2-0792-4cd2-afe7-fbec5c431238 2024-04-24T00:37:15.733 INFO:teuthology.orchestra.run.smithi134.stdout:last_changed 2024-04-24T00:37:15.733390+0000 2024-04-24T00:37:15.733 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-04-24T00:37:15.733390+0000 2024-04-24T00:37:15.733 INFO:teuthology.orchestra.run.smithi134.stdout:min_mon_release 17 (quincy) 2024-04-24T00:37:15.733 INFO:teuthology.orchestra.run.smithi134.stdout:election_strategy: 1 2024-04-24T00:37:15.733 INFO:teuthology.orchestra.run.smithi134.stdout:0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-04-24T00:37:15.733 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-04-24T00:37:15.734 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T00:37:15.781 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID f9adc1f2-0792-4cd2-afe7-fbec5c431238... 2024-04-24T00:37:15.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-24T00:37:15.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-24T00:37:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-04-24T00:37:15.827 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-04-24T00:37:15.827 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T00:37:15.827 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-04-24T00:37:15.827 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.827 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-04-24T00:37:15.827 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T00:37:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.829 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = 172.21.15.134 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on pool no app = False 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 5000 2024-04-24T00:37:15.830 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = f9adc1f2-0792-4cd2-afe7-fbec5c431238 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-04-24T00:37:15.831 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.832 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = stupid 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-04-24T00:37:15.833 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-24T00:37:15.834 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.835 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T00:37:15.836 INFO:teuthology.orchestra.run.smithi067.stdout: debug rbd = 20 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: debug rbd_mirror = 15 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: log to stderr = False 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: rbd default features = 125 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: rbd mirror image policy type = simple 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.0] 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.1] 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.2] 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.837 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.3] 2024-04-24T00:37:15.838 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.838 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.854 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-04-24T00:37:15.855 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-04-24T00:37:15.856 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-04-24T00:37:15.857 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = 172.21.15.134 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: mon client directed command retry = 5 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on pool no app = False 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: ms inject socket failures = 5000 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = f9adc1f2-0792-4cd2-afe7-fbec5c431238 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-04-24T00:37:15.858 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug trim objects = true 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-04-24T00:37:15.859 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: bdev async discard = True 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: bdev enable discard = True 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore allocator = stupid 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-04-24T00:37:15.860 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 1/20 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 1/20 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 4/10 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-04-24T00:37:15.861 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-04-24T00:37:15.862 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-04-24T00:37:15.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd = 20 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd_mirror = 15 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: log to stderr = False 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: rbd default features = 125 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: rbd mirror image policy type = simple 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.865 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-04-24T00:37:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.865 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-04-24T00:37:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.865 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-04-24T00:37:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T00:37:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T00:37:15.866 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-04-24T00:37:15.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-04-24T00:37:15.955 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T00:37:15.955 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:15.955 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-04-24T00:37:15.961 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:15.961 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-04-24T00:37:16.008 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T00:37:16.009 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:37:16.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-24T00:37:16.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-24T00:37:16.033 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:37:16.033 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-24T00:37:16.080 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T00:37:16.080 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:16.080 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-24T00:37:16.080 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-24T00:37:16.105 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:16.105 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-24T00:37:16.153 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T00:37:16.153 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T00:37:16.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-24T00:37:16.198 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-24T00:37:16.201 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T00:37:16.201 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T00:37:16.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-04-24T00:37:16.275 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-04-24T00:37:16.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-04-24T00:37:16.321 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-04-24T00:37:16.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-24T00:37:16.411 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-24T00:37:16.421 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-24T00:37:16.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-24T00:37:16.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-24T00:37:16.654 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-24T00:37:16.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-24T00:37:16.734 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-24T00:37:16.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-24T00:37:16.813 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-24T00:37:16.822 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-24T00:37:16.892 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-24T00:37:16.901 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T00:37:16.901 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T00:37:16.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-04-24T00:37:16.959 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T00:37:16.959 INFO:tasks.ceph:role: cluster2.osd.0 2024-04-24T00:37:16.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T00:37:16.959 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T00:37:17.020 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:37:17.020 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:37:17.020 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:37:17.020 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:37:17.020 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:37:17.020 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:37:17.021 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:37:17.021 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:37:17.021 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:37:17.021 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:37:17.470 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-04-24T00:37:17.472 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-04-24T00:37:17.472 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-04-24T00:37:17.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-04-24T00:37:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:37:17.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:37:17.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-04-24T00:37:17.629 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T00:37:17.629 INFO:tasks.ceph:role: cluster2.osd.1 2024-04-24T00:37:17.629 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T00:37:17.629 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:37:18.160 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-04-24T00:37:18.162 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-04-24T00:37:18.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-04-24T00:37:18.309 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-04-24T00:37:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:37:18.322 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:37:18.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-04-24T00:37:18.377 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T00:37:18.377 INFO:tasks.ceph:role: cluster2.osd.2 2024-04-24T00:37:18.377 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T00:37:18.377 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T00:37:18.440 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:37:18.441 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:37:18.441 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:37:18.441 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:37:18.441 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:37:18.441 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:37:18.441 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:37:18.441 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:37:18.441 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:37:18.441 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:37:18.911 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-04-24T00:37:18.914 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-04-24T00:37:18.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-04-24T00:37:19.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-04-24T00:37:19.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:37:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:37:19.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T00:37:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:19.303+0000 7fae0b2727c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-04-24T00:37:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:19.303+0000 7fae0b2727c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-04-24T00:37:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:19.303+0000 7fae0b2727c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-24T00:37:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:19.303+0000 7fae0b2727c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-24T00:37:19.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:19.303+0000 7fae0b2727c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-24T00:37:19.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:19.303+0000 7fae0b2727c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-04-24T00:37:22.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-04-24T00:37:22.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T00:37:22.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:22.379+0000 7f72d4d337c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-04-24T00:37:22.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:22.379+0000 7f72d4d337c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-04-24T00:37:22.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:22.379+0000 7f72d4d337c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-24T00:37:22.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:22.383+0000 7f72d4d337c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-24T00:37:22.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:22.383+0000 7f72d4d337c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-24T00:37:22.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:22.383+0000 7f72d4d337c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-04-24T00:37:25.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-04-24T00:37:25.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T00:37:25.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:25.499+0000 7f56941a47c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-04-24T00:37:25.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:25.499+0000 7f56941a47c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-04-24T00:37:25.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:25.499+0000 7f56941a47c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-24T00:37:25.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:25.499+0000 7f56941a47c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-24T00:37:25.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:25.499+0000 7f56941a47c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-24T00:37:25.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:25.503+0000 7f56941a47c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-04-24T00:37:28.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-04-24T00:37:28.538 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T00:37:28.538 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:37:28.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-04-24T00:37:28.553 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-04-24T00:37:28.592 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.593 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-04-24T00:37:28.648 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.648 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-04-24T00:37:28.704 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-24T00:37:28.704 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-04-24T00:37:28.710 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.710 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-04-24T00:37:28.753 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.753 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-04-24T00:37:28.802 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.802 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-04-24T00:37:28.849 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.849 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-04-24T00:37:28.897 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.897 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-04-24T00:37:28.945 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.945 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-04-24T00:37:28.993 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:28.993 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-04-24T00:37:29.041 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:29.041 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-04-24T00:37:29.089 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T00:37:29.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo tee -a /etc/ceph/cluster2.keyring 2024-04-24T00:37:29.143 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.x] 2024-04-24T00:37:29.143 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmef/jCxAAxJ9VZDvxXaZsjeTLnWHreg== 2024-04-24T00:37:29.143 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-04-24T00:37:29.143 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA/VChmj/JHEhAAHb4t7w6K/FWV31D1cEzACA== 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBCVChm3MnxFhAAWzzW/uLTNaxow9o+DsXHEg== 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBFVChmKnwPHhAA3s8JyQQq93slSAWqtZ0kRg== 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChm3X17EBAASy8x/62dWNzSTPutllnzJA== 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror] 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChm9r00ExAA006qeC7KrIL11V3i1Oi3zA== 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmAq+PGBAA7PuZa1MKFZAsIWQ5NVULRg== 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmNWGTHRAAN/yfZ4N+Ayc1UdFOslBYJQ== 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-04-24T00:37:29.144 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmB70pIhAAExzZIfDL6GSXcjo7UHLwWg== 2024-04-24T00:37:29.145 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-04-24T00:37:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmjqIPJxAAEQu9qqbzailP750PRPwGNg== 2024-04-24T00:37:29.145 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.4] 2024-04-24T00:37:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmUnXQKxAAtoBewSRkFUSykbrk3pfelg== 2024-04-24T00:37:29.145 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.5] 2024-04-24T00:37:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmjYSGMBAAw5nRtgwPdS0mA0F8ZG+BQg== 2024-04-24T00:37:29.145 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.6] 2024-04-24T00:37:29.145 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmTT1CNRAAnrM9SFd3QwYtzeuf0Y38dg== 2024-04-24T00:37:29.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T00:37:29.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:37:29.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:37:29.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:37:29.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:37:29.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:37:29.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:37:29.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:37:29.753 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:37:29.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:37:29.926 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:37:30.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:37:30.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:37:30.146 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T00:37:30.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-04-24T00:37:30.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-04-24T00:37:30.290 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-04-24T00:37:30.301 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T00:37:30.348 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-04-24T00:37:30.349 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-04-24T00:37:30.349 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-04-24T00:37:30.394 INFO:tasks.ceph.cluster2.mon.a:Started 2024-04-24T00:37:30.394 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-04-24T00:37:30.394 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-04-24T00:37:30.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-04-24T00:37:30.397 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-04-24T00:37:30.397 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T00:37:30.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd crush tunables default 2024-04-24T00:37:30.634 INFO:teuthology.orchestra.run.smithi134.stderr:adjusted tunables profile to default 2024-04-24T00:37:30.652 INFO:tasks.ceph:check_enable_crimson: False 2024-04-24T00:37:30.652 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-04-24T00:37:30.652 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:30.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-04-24T00:37:30.663 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:30.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-04-24T00:37:30.673 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:30.671+0000 7f3f26633180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T00:37:30.713 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:37:30.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-04-24T00:37:30.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd new 1e153e69-c5c5-4e91-bada-a7aedb754f8b 0 2024-04-24T00:37:31.011 INFO:teuthology.orchestra.run.smithi134.stdout:0 2024-04-24T00:37:31.029 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd new 17e7cd39-6768-4e9c-983d-b3cb74060185 1 2024-04-24T00:37:31.081 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:31.079+0000 7f3f26633180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-04-24T00:37:31.226 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:31.223+0000 7f3f26633180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T00:37:31.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd new 72645446-5208-437b-8acb-67f87d544d82 2 2024-04-24T00:37:31.442 INFO:teuthology.orchestra.run.smithi134.stdout:2 2024-04-24T00:37:31.462 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-04-24T00:37:31.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-04-24T00:37:31.464 INFO:tasks.ceph.cluster2.osd.0:Started 2024-04-24T00:37:31.464 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-04-24T00:37:31.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-04-24T00:37:31.467 INFO:tasks.ceph.cluster2.osd.1:Started 2024-04-24T00:37:31.467 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-04-24T00:37:31.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-04-24T00:37:31.469 INFO:tasks.ceph.cluster2.osd.2:Started 2024-04-24T00:37:31.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T00:37:31.666 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:31.663+0000 7f3f26633180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T00:37:31.719 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:31.719+0000 7f3f26633180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T00:37:31.770 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:31.767+0000 7f3f26633180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T00:37:31.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T00:37:31.791+0000 7f51a9595640 0 --2- 172.21.15.134:0/3010591483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f51a40b6810 0x7f51a4143160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-24T00:37:31.848 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:31.847+0000 7f3f26633180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"fsid":"f9adc1f2-0792-4cd2-afe7-fbec5c431238","created":"2024-04-24T00:37:30.525473+0000","modified":"2024-04-24T00:37:31.441935+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T00:37:31.441935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e153e69-c5c5-4e91-bada-a7aedb754f8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17e7cd39-6768-4e9c-983d-b3cb74060185","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72645446-5208-437b-8acb-67f87d544d82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:31.875 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-04-24T00:37:31.875 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T00:37:31.899 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:31.899+0000 7f3f26633180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T00:37:31.975 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:31.975+0000 7f3f26633180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T00:37:32.026 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:32.023+0000 7f3f26633180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T00:37:32.074 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:37:32.071+0000 7f63279c07c0 -1 Falling back to public interface 2024-04-24T00:37:32.093 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:37:32.091+0000 7ff598dac7c0 -1 Falling back to public interface 2024-04-24T00:37:32.094 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:37:32.091+0000 7fbdcf7747c0 -1 Falling back to public interface 2024-04-24T00:37:32.171 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:32.171+0000 7f3f26633180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T00:37:32.177 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-24T00:37:32.225 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:32.223+0000 7f3f26633180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T00:37:32.365 INFO:teuthology.misc.health.smithi134.stdout: 2024-04-24T00:37:32.365 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":5,"fsid":"f9adc1f2-0792-4cd2-afe7-fbec5c431238","created":"2024-04-24T00:37:30.525473+0000","modified":"2024-04-24T00:37:31.441935+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T00:37:31.441935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e153e69-c5c5-4e91-bada-a7aedb754f8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17e7cd39-6768-4e9c-983d-b3cb74060185","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72645446-5208-437b-8acb-67f87d544d82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:32.381 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T00:37:32.450 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-24T00:37:32.450 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-24T00:37:32.450 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr: from numpy import show_config as show_numpy_config 2024-04-24T00:37:32.458 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:32.455+0000 7f3f26633180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T00:37:32.596 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:32.595+0000 7f3f26633180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T00:37:32.710 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:32.707+0000 7f3f26633180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T00:37:32.774 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:32.771+0000 7f3f26633180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T00:37:32.913 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:32.911+0000 7f3f26633180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T00:37:33.415 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:33.415+0000 7f3f26633180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T00:37:33.491 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:33.491+0000 7f3f26633180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T00:37:33.642 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:33.639+0000 7f3f26633180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T00:37:33.718 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:33.715+0000 7f3f26633180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T00:37:34.013 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:34.011+0000 7f3f26633180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T00:37:34.070 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:34.067+0000 7f3f26633180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T00:37:35.794 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:37:35.791+0000 7f63279c07c0 -1 osd.0 0 log_to_monitors true 2024-04-24T00:37:35.884 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:37:35.879+0000 7fbdcf7747c0 -1 osd.2 0 log_to_monitors true 2024-04-24T00:37:35.892 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:37:35.887+0000 7ff598dac7c0 -1 osd.1 0 log_to_monitors true 2024-04-24T00:37:36.098 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-04-24T00:37:36.095+0000 7f3ecda79640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-24T00:37:38.684 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-24T00:37:39.112 INFO:teuthology.misc.health.smithi134.stdout: 2024-04-24T00:37:39.113 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":7,"fsid":"f9adc1f2-0792-4cd2-afe7-fbec5c431238","created":"2024-04-24T00:37:30.525473+0000","modified":"2024-04-24T00:37:37.154022+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T00:37:31.441935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e153e69-c5c5-4e91-bada-a7aedb754f8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17e7cd39-6768-4e9c-983d-b3cb74060185","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72645446-5208-437b-8acb-67f87d544d82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:39.136 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:37:39.131+0000 7fbdcaefa640 -1 osd.2 0 waiting for initial osdmap 2024-04-24T00:37:39.136 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:37:39.135+0000 7f6323146640 -1 osd.0 0 waiting for initial osdmap 2024-04-24T00:37:39.138 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T00:37:39.169 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:37:39.167+0000 7f631b78d640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:37:39.169 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:37:39.167+0000 7fbdc3541640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:37:39.191 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:37:39.187+0000 7ff595d68640 -1 osd.1 0 waiting for initial osdmap 2024-04-24T00:37:39.213 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:37:39.211+0000 7ff58cb79640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:37:45.441 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-24T00:37:45.735 INFO:teuthology.misc.health.smithi134.stdout: 2024-04-24T00:37:45.735 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"f9adc1f2-0792-4cd2-afe7-fbec5c431238","created":"2024-04-24T00:37:30.525473+0000","modified":"2024-04-24T00:37:44.394689+0000","last_up_change":"2024-04-24T00:37:40.143248+0000","last_in_change":"2024-04-24T00:37:31.441935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:37:42.173096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e153e69-c5c5-4e91-bada-a7aedb754f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6801","nonce":2330099020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6803","nonce":2330099020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6807","nonce":2330099020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6805","nonce":2330099020}]},"public_addr":"172.21.15.134:6801/2330099020","cluster_addr":"172.21.15.134:6803/2330099020","heartbeat_back_addr":"172.21.15.134:6807/2330099020","heartbeat_front_addr":"172.21.15.134:6805/2330099020","state":["exists","up"]},{"osd":1,"uuid":"17e7cd39-6768-4e9c-983d-b3cb74060185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6809","nonce":3497594701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6811","nonce":3497594701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6815","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6817","nonce":3497594701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6814","nonce":3497594701}]},"public_addr":"172.21.15.134:6809/3497594701","cluster_addr":"172.21.15.134:6811/3497594701","heartbeat_back_addr":"172.21.15.134:6817/3497594701","heartbeat_front_addr":"172.21.15.134:6814/3497594701","state":["exists","up"]},{"osd":2,"uuid":"72645446-5208-437b-8acb-67f87d544d82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6813","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6816","nonce":101056980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6819","nonce":101056980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6823","nonce":101056980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6821","nonce":101056980}]},"public_addr":"172.21.15.134:6816/101056980","cluster_addr":"172.21.15.134:6819/101056980","heartbeat_back_addr":"172.21.15.134:6823/101056980","heartbeat_front_addr":"172.21.15.134:6821/101056980","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.788520+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.877477+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.924657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:45.753 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-24T00:37:45.753 INFO:tasks.ceph:Creating RBD pool 2024-04-24T00:37:45.753 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-04-24T00:37:46.406 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'rbd' created 2024-04-24T00:37:46.433 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T00:37:47.407 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T00:37:47.436 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-04-24T00:37:47.436 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-04-24T00:37:47.437 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-04-24T00:37:47.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-04-24T00:37:47.437 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T00:37:47.776 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:47.794 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2153108470},{"type":"v1","addr":"172.21.15.67:6827","nonce":2153108470}]},"active_addr":"172.21.15.67:6827/2153108470","active_change":"2024-04-24T00:37:34.089826+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3765154455}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":492635232}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":252010739}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1234185219}]}]} 2024-04-24T00:37:47.797 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-04-24T00:37:47.797 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-04-24T00:37:47.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T00:37:48.094 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:48.094 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":14,"fsid":"f9adc1f2-0792-4cd2-afe7-fbec5c431238","created":"2024-04-24T00:37:30.525473+0000","modified":"2024-04-24T00:37:47.407087+0000","last_up_change":"2024-04-24T00:37:40.143248+0000","last_in_change":"2024-04-24T00:37:31.441935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:37:42.173096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:37:46.090758+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e153e69-c5c5-4e91-bada-a7aedb754f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6801","nonce":2330099020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6803","nonce":2330099020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6807","nonce":2330099020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6805","nonce":2330099020}]},"public_addr":"172.21.15.134:6801/2330099020","cluster_addr":"172.21.15.134:6803/2330099020","heartbeat_back_addr":"172.21.15.134:6807/2330099020","heartbeat_front_addr":"172.21.15.134:6805/2330099020","state":["exists","up"]},{"osd":1,"uuid":"17e7cd39-6768-4e9c-983d-b3cb74060185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6809","nonce":3497594701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6811","nonce":3497594701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6815","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6817","nonce":3497594701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6814","nonce":3497594701}]},"public_addr":"172.21.15.134:6809/3497594701","cluster_addr":"172.21.15.134:6811/3497594701","heartbeat_back_addr":"172.21.15.134:6817/3497594701","heartbeat_front_addr":"172.21.15.134:6814/3497594701","state":["exists","up"]},{"osd":2,"uuid":"72645446-5208-437b-8acb-67f87d544d82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6813","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6816","nonce":101056980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6819","nonce":101056980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6823","nonce":101056980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6821","nonce":101056980}]},"public_addr":"172.21.15.134:6816/101056980","cluster_addr":"172.21.15.134:6819/101056980","heartbeat_back_addr":"172.21.15.134:6823/101056980","heartbeat_front_addr":"172.21.15.134:6821/101056980","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.788520+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.877477+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.924657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:48.114 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-04-24T00:37:48.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T00:37:48.405 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:48.405 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":14,"fsid":"f9adc1f2-0792-4cd2-afe7-fbec5c431238","created":"2024-04-24T00:37:30.525473+0000","modified":"2024-04-24T00:37:47.407087+0000","last_up_change":"2024-04-24T00:37:40.143248+0000","last_in_change":"2024-04-24T00:37:31.441935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:37:42.173096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:37:46.090758+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e153e69-c5c5-4e91-bada-a7aedb754f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6801","nonce":2330099020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6803","nonce":2330099020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6807","nonce":2330099020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6805","nonce":2330099020}]},"public_addr":"172.21.15.134:6801/2330099020","cluster_addr":"172.21.15.134:6803/2330099020","heartbeat_back_addr":"172.21.15.134:6807/2330099020","heartbeat_front_addr":"172.21.15.134:6805/2330099020","state":["exists","up"]},{"osd":1,"uuid":"17e7cd39-6768-4e9c-983d-b3cb74060185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6809","nonce":3497594701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6811","nonce":3497594701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6815","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6817","nonce":3497594701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6814","nonce":3497594701}]},"public_addr":"172.21.15.134:6809/3497594701","cluster_addr":"172.21.15.134:6811/3497594701","heartbeat_back_addr":"172.21.15.134:6817/3497594701","heartbeat_front_addr":"172.21.15.134:6814/3497594701","state":["exists","up"]},{"osd":2,"uuid":"72645446-5208-437b-8acb-67f87d544d82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6813","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6816","nonce":101056980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6819","nonce":101056980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6823","nonce":101056980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6821","nonce":101056980}]},"public_addr":"172.21.15.134:6816/101056980","cluster_addr":"172.21.15.134:6819/101056980","heartbeat_back_addr":"172.21.15.134:6823/101056980","heartbeat_front_addr":"172.21.15.134:6821/101056980","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.788520+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.877477+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.924657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:37:48.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-04-24T00:37:48.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-04-24T00:37:48.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-04-24T00:37:48.576 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-04-24T00:37:48.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-24T00:37:48.587 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-04-24T00:37:48.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-24T00:37:48.599 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T00:37:48.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-24T00:37:48.855 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T00:37:48.872 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-04-24T00:37:48.924 INFO:teuthology.orchestra.run.smithi134.stdout:34359738371 2024-04-24T00:37:48.929 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T00:37:48.944 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.0 2024-04-24T00:37:48.949 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-04-24T00:37:49.873 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-24T00:37:49.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-24T00:37:49.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-24T00:37:50.189 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T00:37:50.207 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-04-24T00:37:50.266 INFO:teuthology.orchestra.run.smithi134.stdout:34359738371 2024-04-24T00:37:50.284 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.0 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T00:37:50.314 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-04-24T00:37:51.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-24T00:37:51.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-24T00:37:51.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-24T00:37:51.531 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-04-24T00:37:51.549 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2024-04-24T00:37:51.549 DEBUG:teuthology.parallel:result is None 2024-04-24T00:37:51.582 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T00:37:51.599 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.0 2024-04-24T00:37:51.600 DEBUG:teuthology.parallel:result is None 2024-04-24T00:37:51.615 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-04-24T00:37:51.632 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.1 2024-04-24T00:37:51.632 DEBUG:teuthology.parallel:result is None 2024-04-24T00:37:51.632 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-24T00:37:51.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T00:37:51.958 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:51.959 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T00:37:51.975 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-24T00:37:50.099851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82968,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032552,"statfs":{"total":289910292480,"available":289825333248,"internally_reserved":0,"allocated":1556480,"data_stored":1282004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.428224+0000","last_change":"2024-04-24T00:37:47.429755+0000","last_active":"2024-04-24T00:37:48.428224+0000","last_peered":"2024-04-24T00:37:48.428224+0000","last_clean":"2024-04-24T00:37:48.428224+0000","last_became_active":"2024-04-24T00:37:47.429213+0000","last_became_peered":"2024-04-24T00:37:47.429213+0000","last_unstale":"2024-04-24T00:37:48.428224+0000","last_undegraded":"2024-04-24T00:37:48.428224+0000","last_fullsized":"2024-04-24T00:37:48.428224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:34:19.765148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.427882+0000","last_change":"2024-04-24T00:37:47.427816+0000","last_active":"2024-04-24T00:37:48.427882+0000","last_peered":"2024-04-24T00:37:48.427882+0000","last_clean":"2024-04-24T00:37:48.427882+0000","last_became_active":"2024-04-24T00:37:47.427410+0000","last_became_peered":"2024-04-24T00:37:47.427410+0000","last_unstale":"2024-04-24T00:37:48.427882+0000","last_undegraded":"2024-04-24T00:37:48.427882+0000","last_fullsized":"2024-04-24T00:37:48.427882+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:05:01.512170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.427883+0000","last_change":"2024-04-24T00:37:47.428837+0000","last_active":"2024-04-24T00:37:48.427883+0000","last_peered":"2024-04-24T00:37:48.427883+0000","last_clean":"2024-04-24T00:37:48.427883+0000","last_became_active":"2024-04-24T00:37:47.428266+0000","last_became_peered":"2024-04-24T00:37:47.428266+0000","last_unstale":"2024-04-24T00:37:48.427883+0000","last_undegraded":"2024-04-24T00:37:48.427883+0000","last_fullsized":"2024-04-24T00:37:48.427883+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:43.968711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.429879+0000","last_change":"2024-04-24T00:37:47.429020+0000","last_active":"2024-04-24T00:37:48.429879+0000","last_peered":"2024-04-24T00:37:48.429879+0000","last_clean":"2024-04-24T00:37:48.429879+0000","last_became_active":"2024-04-24T00:37:47.427471+0000","last_became_peered":"2024-04-24T00:37:47.427471+0000","last_unstale":"2024-04-24T00:37:48.429879+0000","last_undegraded":"2024-04-24T00:37:48.429879+0000","last_fullsized":"2024-04-24T00:37:48.429879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:41.046954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T00:37:47.433837+0000","last_change":"2024-04-24T00:37:47.427475+0000","last_active":"2024-04-24T00:37:47.433837+0000","last_peered":"2024-04-24T00:37:47.433837+0000","last_clean":"2024-04-24T00:37:47.433837+0000","last_became_active":"2024-04-24T00:37:47.426947+0000","last_became_peered":"2024-04-24T00:37:47.426947+0000","last_unstale":"2024-04-24T00:37:47.433837+0000","last_undegraded":"2024-04-24T00:37:47.433837+0000","last_fullsized":"2024-04-24T00:37:47.433837+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:15.620870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T00:37:47.434273+0000","last_change":"2024-04-24T00:37:47.431766+0000","last_active":"2024-04-24T00:37:47.434273+0000","last_peered":"2024-04-24T00:37:47.434273+0000","last_clean":"2024-04-24T00:37:47.434273+0000","last_became_active":"2024-04-24T00:37:47.431433+0000","last_became_peered":"2024-04-24T00:37:47.431433+0000","last_unstale":"2024-04-24T00:37:47.434273+0000","last_undegraded":"2024-04-24T00:37:47.434273+0000","last_fullsized":"2024-04-24T00:37:47.434273+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:24:14.658521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T00:37:47.423907+0000","last_change":"2024-04-24T00:37:47.422610+0000","last_active":"2024-04-24T00:37:47.423907+0000","last_peered":"2024-04-24T00:37:47.423907+0000","last_clean":"2024-04-24T00:37:47.423907+0000","last_became_active":"2024-04-24T00:37:47.422197+0000","last_became_peered":"2024-04-24T00:37:47.422197+0000","last_unstale":"2024-04-24T00:37:47.423907+0000","last_undegraded":"2024-04-24T00:37:47.423907+0000","last_fullsized":"2024-04-24T00:37:47.423907+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:14:48.132507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.428016+0000","last_change":"2024-04-24T00:37:47.430799+0000","last_active":"2024-04-24T00:37:48.428016+0000","last_peered":"2024-04-24T00:37:48.428016+0000","last_clean":"2024-04-24T00:37:48.428016+0000","last_became_active":"2024-04-24T00:37:47.430316+0000","last_became_peered":"2024-04-24T00:37:47.430316+0000","last_unstale":"2024-04-24T00:37:48.428016+0000","last_undegraded":"2024-04-24T00:37:48.428016+0000","last_fullsized":"2024-04-24T00:37:48.428016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:25:37.002437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.427647+0000","last_change":"2024-04-24T00:37:43.400334+0000","last_active":"2024-04-24T00:37:48.427647+0000","last_peered":"2024-04-24T00:37:48.427647+0000","last_clean":"2024-04-24T00:37:48.427647+0000","last_became_active":"2024-04-24T00:37:43.399665+0000","last_became_peered":"2024-04-24T00:37:43.399665+0000","last_unstale":"2024-04-24T00:37:48.427647+0000","last_undegraded":"2024-04-24T00:37:48.427647+0000","last_fullsized":"2024-04-24T00:37:48.427647+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_clean_scrub_stamp":"2024-04-24T00:37:42.383719+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:35:49.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":716800,"data_stored":624124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":716800,"data_stored":624124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:37:51.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T00:37:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:52.255 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T00:37:52.271 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-24T00:37:52.100538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82608,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032912,"statfs":{"total":289910292480,"available":289825701888,"internally_reserved":0,"allocated":1581056,"data_stored":1292438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001758"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.428224+0000","last_change":"2024-04-24T00:37:47.429755+0000","last_active":"2024-04-24T00:37:48.428224+0000","last_peered":"2024-04-24T00:37:48.428224+0000","last_clean":"2024-04-24T00:37:48.428224+0000","last_became_active":"2024-04-24T00:37:47.429213+0000","last_became_peered":"2024-04-24T00:37:47.429213+0000","last_unstale":"2024-04-24T00:37:48.428224+0000","last_undegraded":"2024-04-24T00:37:48.428224+0000","last_fullsized":"2024-04-24T00:37:48.428224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:34:19.765148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.427882+0000","last_change":"2024-04-24T00:37:47.427816+0000","last_active":"2024-04-24T00:37:48.427882+0000","last_peered":"2024-04-24T00:37:48.427882+0000","last_clean":"2024-04-24T00:37:48.427882+0000","last_became_active":"2024-04-24T00:37:47.427410+0000","last_became_peered":"2024-04-24T00:37:47.427410+0000","last_unstale":"2024-04-24T00:37:48.427882+0000","last_undegraded":"2024-04-24T00:37:48.427882+0000","last_fullsized":"2024-04-24T00:37:48.427882+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:05:01.512170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.427883+0000","last_change":"2024-04-24T00:37:47.428837+0000","last_active":"2024-04-24T00:37:48.427883+0000","last_peered":"2024-04-24T00:37:48.427883+0000","last_clean":"2024-04-24T00:37:48.427883+0000","last_became_active":"2024-04-24T00:37:47.428266+0000","last_became_peered":"2024-04-24T00:37:47.428266+0000","last_unstale":"2024-04-24T00:37:48.427883+0000","last_undegraded":"2024-04-24T00:37:48.427883+0000","last_fullsized":"2024-04-24T00:37:48.427883+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:43.968711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.429879+0000","last_change":"2024-04-24T00:37:47.429020+0000","last_active":"2024-04-24T00:37:48.429879+0000","last_peered":"2024-04-24T00:37:48.429879+0000","last_clean":"2024-04-24T00:37:48.429879+0000","last_became_active":"2024-04-24T00:37:47.427471+0000","last_became_peered":"2024-04-24T00:37:47.427471+0000","last_unstale":"2024-04-24T00:37:48.429879+0000","last_undegraded":"2024-04-24T00:37:48.429879+0000","last_fullsized":"2024-04-24T00:37:48.429879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:41.046954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.600383+0000","last_change":"2024-04-24T00:37:47.427475+0000","last_active":"2024-04-24T00:37:48.600383+0000","last_peered":"2024-04-24T00:37:48.600383+0000","last_clean":"2024-04-24T00:37:48.600383+0000","last_became_active":"2024-04-24T00:37:47.426947+0000","last_became_peered":"2024-04-24T00:37:47.426947+0000","last_unstale":"2024-04-24T00:37:48.600383+0000","last_undegraded":"2024-04-24T00:37:48.600383+0000","last_fullsized":"2024-04-24T00:37:48.600383+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:09:53.202558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.600477+0000","last_change":"2024-04-24T00:37:47.431766+0000","last_active":"2024-04-24T00:37:48.600477+0000","last_peered":"2024-04-24T00:37:48.600477+0000","last_clean":"2024-04-24T00:37:48.600477+0000","last_became_active":"2024-04-24T00:37:47.431433+0000","last_became_peered":"2024-04-24T00:37:47.431433+0000","last_unstale":"2024-04-24T00:37:48.600477+0000","last_undegraded":"2024-04-24T00:37:48.600477+0000","last_fullsized":"2024-04-24T00:37:48.600477+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:29:01.749521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.600411+0000","last_change":"2024-04-24T00:37:47.422610+0000","last_active":"2024-04-24T00:37:48.600411+0000","last_peered":"2024-04-24T00:37:48.600411+0000","last_clean":"2024-04-24T00:37:48.600411+0000","last_became_active":"2024-04-24T00:37:47.422197+0000","last_became_peered":"2024-04-24T00:37:47.422197+0000","last_unstale":"2024-04-24T00:37:48.600411+0000","last_undegraded":"2024-04-24T00:37:48.600411+0000","last_fullsized":"2024-04-24T00:37:48.600411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:02:16.063327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.428016+0000","last_change":"2024-04-24T00:37:47.430799+0000","last_active":"2024-04-24T00:37:48.428016+0000","last_peered":"2024-04-24T00:37:48.428016+0000","last_clean":"2024-04-24T00:37:48.428016+0000","last_became_active":"2024-04-24T00:37:47.430316+0000","last_became_peered":"2024-04-24T00:37:47.430316+0000","last_unstale":"2024-04-24T00:37:48.428016+0000","last_undegraded":"2024-04-24T00:37:48.428016+0000","last_fullsized":"2024-04-24T00:37:48.428016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:25:37.002437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.427647+0000","last_change":"2024-04-24T00:37:43.400334+0000","last_active":"2024-04-24T00:37:48.427647+0000","last_peered":"2024-04-24T00:37:48.427647+0000","last_clean":"2024-04-24T00:37:48.427647+0000","last_became_active":"2024-04-24T00:37:43.399665+0000","last_became_peered":"2024-04-24T00:37:43.399665+0000","last_unstale":"2024-04-24T00:37:48.427647+0000","last_undegraded":"2024-04-24T00:37:48.427647+0000","last_fullsized":"2024-04-24T00:37:48.427647+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_clean_scrub_stamp":"2024-04-24T00:37:42.383719+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:35:49.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:37:52.272 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-24T00:37:52.272 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-04-24T00:37:52.272 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-04-24T00:37:52.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-04-24T00:37:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T00:37:52.580 INFO:teuthology.orchestra.run.smithi134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T00:37:52.597 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-04-24T00:37:52.598 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T00:37:52.609 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T00:37:52.609 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T00:37:52.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:52.955 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror] 2024-04-24T00:37:52.955 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAXVChmRKiIIhAAldBUuPvYcu9e1TP8PhOoog== 2024-04-24T00:37:52.955 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror-peer" 2024-04-24T00:37:52.955 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:52.955 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror 2024-04-24T00:37:52.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:53.316 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-04-24T00:37:53.317 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAXVChmgZhlKBAAbeHibthsjhfWhU1qhLcAqA== 2024-04-24T00:37:53.317 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror" 2024-04-24T00:37:53.317 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:53.317 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.0 2024-04-24T00:37:53.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:53.702 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-04-24T00:37:53.702 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAXVChmWKpILRAACSglusKKt6eWUdY5A96mfA== 2024-04-24T00:37:53.703 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror" 2024-04-24T00:37:53.703 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:53.703 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.1 2024-04-24T00:37:53.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAXVChmRPqKMhAAb7x2QliTCYuJm6zDkDnxfw== 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror" 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.2 2024-04-24T00:37:54.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:54.458 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-04-24T00:37:54.458 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAXVChmYQunNxAAYAzk15KY1yveX64xvruefw== 2024-04-24T00:37:54.458 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror" 2024-04-24T00:37:54.458 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:54.458 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.3 2024-04-24T00:37:54.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:54.824 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror] 2024-04-24T00:37:54.824 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChm9r00ExAA006qeC7KrIL11V3i1Oi3zA== 2024-04-24T00:37:54.824 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror-peer" 2024-04-24T00:37:54.824 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:54.824 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror 2024-04-24T00:37:54.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:55.190 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-04-24T00:37:55.190 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmAq+PGBAA7PuZa1MKFZAsIWQ5NVULRg== 2024-04-24T00:37:55.190 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror" 2024-04-24T00:37:55.190 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:55.191 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.0 2024-04-24T00:37:55.209 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:55.561 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-04-24T00:37:55.561 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmNWGTHRAAN/yfZ4N+Ayc1UdFOslBYJQ== 2024-04-24T00:37:55.561 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror" 2024-04-24T00:37:55.562 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:55.562 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.1 2024-04-24T00:37:55.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:55.933 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-04-24T00:37:55.934 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmB70pIhAAExzZIfDL6GSXcjo7UHLwWg== 2024-04-24T00:37:55.934 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror" 2024-04-24T00:37:55.934 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:55.934 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.2 2024-04-24T00:37:55.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T00:37:56.303 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-04-24T00:37:56.303 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8VChmjqIPJxAAEQu9qqbzailP750PRPwGNg== 2024-04-24T00:37:56.303 INFO:teuthology.orchestra.run.smithi134.stdout: caps mon = "profile rbd-mirror" 2024-04-24T00:37:56.303 INFO:teuthology.orchestra.run.smithi134.stdout: caps osd = "profile rbd" 2024-04-24T00:37:56.304 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.3 2024-04-24T00:37:56.322 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T00:37:56.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T00:37:56.339 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T00:37:56.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T00:37:56.342 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T00:37:56.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T00:37:56.358 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T00:37:56.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T00:37:56.361 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T00:37:56.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T00:37:56.392 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T00:37:56.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T00:37:56.395 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T00:37:56.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-24T00:37:56.426 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-24T00:37:56.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-24T00:37:56.431 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T00:37:56.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T00:37:56.460 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T00:37:56.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T00:37:56.462 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T00:37:56.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T00:37:56.492 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T00:37:56.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T00:37:56.495 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T00:37:56.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-24T00:37:56.526 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-24T00:37:56.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-24T00:37:56.528 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T00:37:56.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-24T00:37:56.561 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-24T00:37:56.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-24T00:37:56.563 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-24T00:37:56.594 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1713919076 2024-04-24T00:37:56.594 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-24T00:37:56.594 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-24T00:37:56.626 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1713919076 2024-04-24T00:37:56.626 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-24T00:37:56.626 INFO:teuthology.run_tasks:Running task workunit... 2024-04-24T00:37:56.663 INFO:tasks.workunit:Pulling workunits from ref b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:37:56.663 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-24T00:37:56.663 DEBUG:teuthology.orchestra.run.smithi134:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T00:37:56.664 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-04-24T00:37:56.664 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-04-24T00:37:56.664 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-04-24T00:37:56.664 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-04-24T00:37:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:37:56.670 INFO:teuthology.orchestra.run.smithi134.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-04-24T00:37:56.670 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T00:37:56.718 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T00:37:56.718 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-04-24T00:37:56.765 INFO:tasks.workunit:timeout=6h 2024-04-24T00:37:56.765 INFO:tasks.workunit:cleanup=True 2024-04-24T00:37:56.765 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:37:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-04-24T00:39:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Note: switching to 'b98aab6c00f483888c049d0d8e7bc507e115dd86'. 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: git switch -c 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T00:39:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Or undo this operation with: 2024-04-24T00:39:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T00:39:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: git switch - 2024-04-24T00:39:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T00:39:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T00:39:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T00:39:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:HEAD is now at b98aab6c00f Merge pull request #57047 from zdover23/wip-doc-2024-04-23-backport-57032-to-squid 2024-04-24T00:39:05.628 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-04-24T00:39:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-24T00:39:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-24T00:39:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-24T00:39:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-24T00:39:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-24T00:39:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-24T00:39:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-24T00:39:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-24T00:39:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-24T00:39:06.628 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T00:39:06.629 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-04-24T00:39:06.678 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-04-24T00:39:06.680 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-04-24T00:39:06.680 DEBUG:teuthology.orchestra.run.smithi134:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b98aab6c00f483888c049d0d8e7bc507e115dd86 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=pool RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=journal RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-24T00:39:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ IMAGE_COUNT=50 2024-04-24T00:39:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ export LOCKDEP=0 2024-04-24T00:39:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ LOCKDEP=0 2024-04-24T00:39:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-24T00:39:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-04-24T00:39:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ type xmlstarlet 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ XMLSTARLET=xmlstarlet 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ RBD_MIRROR_INSTANCES=4 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CLUSTER1=cluster1 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CLUSTER2=cluster2 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ PEER_CLUSTER_SUFFIX= 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ POOL=mirror 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ PARENT_POOL=mirror_parent 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ NS1=ns1 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ NS2=ns2 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ TEMPDIR= 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ID=mirror 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-04-24T00:39:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ RBD_MIRROR_MODE=journal 2024-04-24T00:39:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ MIRROR_POOL_MODE=pool 2024-04-24T00:39:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' journal = snapshot ']' 2024-04-24T00:39:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-04-24T00:39:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS='--id mirror' 2024-04-24T00:39:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ LAST_MIRROR_INSTANCE=3 2024-04-24T00:39:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-24T00:39:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-04-24T00:39:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-04-24T00:39:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_BIN=. 2024-04-24T00:39:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_SRC=. 2024-04-24T00:39:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' -e CMakeCache.txt ']' 2024-04-24T00:39:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 -gt 0 ']' 2024-04-24T00:39:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ setup 2024-04-24T00:39:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local c 2024-04-24T00:39:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-04-24T00:39:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ setup_tempdir 2024-04-24T00:39:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mktemp -d 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ TEMPDIR=/tmp/tmp.aEE7yWfHIo 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z 1 ']' 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ setup_pools cluster1 cluster2 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster2 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mon_map_file 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mon_addr 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local admin_key_file 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-04-24T00:39:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror' already exists 2024-04-24T00:39:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-04-24T00:39:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror_parent' already exists 2024-04-24T00:39:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 pool init mirror 2024-04-24T00:39:11.665 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.43907941884204404) > weight (0.25) 2024-04-24T00:39:11.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6906648658752839) > weight (0.25) 2024-04-24T00:39:11.666 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-24T00:39:11.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-24T00:39:11.667 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-24T00:39:11.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T00:39:11.663+0000 7ff964965640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 29653) UID: 1000 2024-04-24T00:39:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-04-24T00:39:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2024-04-24T00:39:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-04-24T00:39:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-04-24T00:39:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-04-24T00:39:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2024-04-24T00:39:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-04-24T00:39:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z ']' 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z ']' 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_add cluster1 mirror cluster2 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local client_cluster=cluster2 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster2 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid_var_name 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code 2024-04-24T00:39:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local peer_uuid 2024-04-24T00:39:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T00:39:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:39:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-04-24T00:39:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_uuid=bffcfff3-03be-451e-b6e0-2073d9715cd3 2024-04-24T00:39:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T00:39:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:39:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T00:39:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 -eq 0 2024-04-24T00:39:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror_parent 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local client_cluster=cluster2 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster2 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid_var_name 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local peer_uuid 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_uuid=c339ce01-cdec-47ae-aac0-75ced2772655 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 -eq 0 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ setup_pools cluster2 cluster1 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:39:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster1 2024-04-24T00:39:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mon_map_file 2024-04-24T00:39:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mon_addr 2024-04-24T00:39:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local admin_key_file 2024-04-24T00:39:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid 2024-04-24T00:39:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-04-24T00:39:15.667 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-24T00:39:15.667 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:39:15.668 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-24T00:39:15.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T00:39:15.668 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T00:39:15.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T00:39:15.670 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-04-24T00:39:15.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T00:39:15.799+0000 7f0a9b4c6880 -1 rbd::mirror::PoolReplayer: 0x557e012fbb00 init: error accessing remote pool mirror: (2) No such file or directory 2024-04-24T00:39:15.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T00:39:15.867+0000 7f0a9b4c6880 -1 rbd::mirror::PoolReplayer: 0x557e02394600 init: error accessing remote pool mirror_parent: (2) No such file or directory 2024-04-24T00:39:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror' already exists 2024-04-24T00:39:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-04-24T00:39:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror_parent' already exists 2024-04-24T00:39:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 pool init mirror 2024-04-24T00:39:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-04-24T00:39:21.665 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2749370394284041) > weight (0.25) 2024-04-24T00:39:21.666 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T00:39:21.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T00:39:21.666 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-24T00:39:21.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T00:39:21.663+0000 7fb50bcbb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 29750) UID: 1000 2024-04-24T00:39:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T00:39:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2024-04-24T00:39:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-04-24T00:39:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-04-24T00:39:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-04-24T00:39:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2024-04-24T00:39:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-04-24T00:39:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z ']' 2024-04-24T00:39:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z ']' 2024-04-24T00:39:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_add cluster2 mirror cluster1 2024-04-24T00:39:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:39:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local client_cluster=cluster1 2024-04-24T00:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster1 2024-04-24T00:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid_var_name 2024-04-24T00:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code 2024-04-24T00:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local peer_uuid 2024-04-24T00:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T00:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-04-24T00:39:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_uuid=45c98711-6a39-475f-9b67-8ecb090f21ed 2024-04-24T00:39:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T00:39:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:39:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T00:39:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 -eq 0 2024-04-24T00:39:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror_parent 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local client_cluster=cluster1 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster1 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid_var_name 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local peer_uuid 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T00:39:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:39:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-04-24T00:39:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_uuid=f4339dd1-0704-47b8-b0b1-32d36923a636 2024-04-24T00:39:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T00:39:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:39:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T00:39:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 -eq 0 2024-04-24T00:39:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:39:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirrors cluster1 2024-04-24T00:39:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1:0 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:39:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1:1 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1:2 2024-04-24T00:39:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T00:39:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1:3 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:39:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirrors cluster2 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster2:0 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:0 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster2:0 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:39:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster2 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster2:0 ']' 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster2 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster2 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster2:1 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:1 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-04-24T00:39:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster2:1 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster2 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster2:1 ']' 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster2 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster2 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster2:2 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:2 2024-04-24T00:39:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster2:2 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster2 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster2:2 ']' 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster2 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster2 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T00:39:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster2:3 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:3 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster2:3 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster2 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster2:3 ']' 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster2 2024-04-24T00:39:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster2 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.aEE7yWfHIo/rbd-mirror.test.log 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 00:39:24 TEST: add image and test replay after client crashes 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 00:39:24 TEST: add image and test replay after client crashes 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=test 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 512M 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror test 512M 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:39:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T00:39:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T00:39:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:39:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/test 2024-04-24T00:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:39:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:39:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:39:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:39:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:39:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:39:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:39:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:39:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:39:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:39:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:39:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:39:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:39:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:39:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:39:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:39:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:39:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:39:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:39:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:39:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:39:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:39:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:39:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:39:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:39:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:39:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:39:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:39:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:39:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:39:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:39:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:39:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:39:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:39:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:39:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:39:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:39:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:39:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:39:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:39:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:39:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:39:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:39:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:39:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:39:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:39:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:39:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:39:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:39:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:39:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:39:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:39:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:39:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:39:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:39:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:39:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:39:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:39:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:39:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:39:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:39:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:39:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 1 2024-04-24T00:39:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:39:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:39:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:39:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:39:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:39:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:39:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:39:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:39:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:39:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:39:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:39:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:39:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:39:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:39:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:39:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:39:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:39:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:39:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:39:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:39:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:39:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:39:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:39:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:39:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:39:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:39:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:39:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:39:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:39:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:39:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:39:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:39:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:39:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:39:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:39:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:39:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:39:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:39:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:39:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:39:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:39:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:39:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:39:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 2 2024-04-24T00:39:27.667 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T00:39:27.667 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:39:27.667 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T00:39:27.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T00:39:27.668 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T00:39:27.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T00:39:27.671 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-04-24T00:39:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:39:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:39:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:39:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:39:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:39:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:39:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:39:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:39:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:39:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:39:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:39:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:39:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:39:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:39:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:39:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:39:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:39:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:39:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:39:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:39:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:39:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:39:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:39:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:39:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:39:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:39:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:39:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:39:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:39:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:39:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:39:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:39:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:39:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:39:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:39:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:39:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:39:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:39:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:39:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:39:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:39:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:39:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:39:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:39:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:39:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:39:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:39:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:39:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:39:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:39:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:39:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:39:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:39:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:39:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:39:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:39:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:39:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:39:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:39:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 4 2024-04-24T00:39:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:39:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:39:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:39:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:39:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:39:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:39:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:39:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:39:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:39:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:39:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:39:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:39:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:39:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:39:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:39:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:39:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:39:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:39:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name= 2024-04-24T00:39:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 10 2024-04-24T00:39:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:39:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:39:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:39:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:39:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=8 2024-04-24T00:39:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:39:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:39:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 16763904 bytes 2024-04-24T00:39:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 96389120 bytes 2024-04-24T00:39:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 150748160 bytes 2024-04-24T00:39:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 170939392 bytes 2024-04-24T00:39:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 190751744 bytes 2024-04-24T00:39:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 215200768 bytes 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:39:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:39:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:39:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:39:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:39:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:39:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:39:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:39:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:39:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:39:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.0 on smithi134 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:39:30 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:39:28' 2024-04-24T00:39:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:39:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:39:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:39:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-04-24T00:39:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:39:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:39:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:39:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:39:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:39:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.0 on smithi134 2024-04-24T00:39:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:39:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:39:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:39:30' 2024-04-24T00:39:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:39:30' 2024-04-24T00:39:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:39:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:39:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:39:30' 2024-04-24T00:39:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:39:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:39:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:39:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='WARNINGWARNINGOK142094270mirror.0smithi13419.0.0-2424-gb98aab6cfalseOK42114271mirror.1smithi13419.0.0-2424-gb98aab6ctrueOK42194274mirror.3smithi13419.0.0-2424-gb98aab6cfalseOK4259mirror.2smithi13419.0.0-2424-gb98aab6cfalseWARNINGunable to access remote pooltestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42094270mirror.0smithi1342024-04-24 00:39:30cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:39:28' 2024-04-24T00:39:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:39:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:39:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:39:30' 2024-04-24T00:39:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:39:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:39:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:39:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:39:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:39:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:39:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:39:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1 2024-04-24T00:39:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap1 2024-04-24T00:39:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:39:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:39:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:39:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1 2024-04-24T00:39:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:39:45.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T00:39:45.878+0000 7f0a9b4c6880 -1 rbd::mirror::Mirror: 0x557e01313e60 update_pool_replayers: restarting failed pool replayer for uuid: bffcfff3-03be-451e-b6e0-2073d9715cd3 cluster: cluster2 client: client.mirror 2024-04-24T00:39:45.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T00:39:45.966+0000 7f0a9b4c6880 -1 rbd::mirror::Mirror: 0x557e01313e60 update_pool_replayers: restarting failed pool replayer for uuid: c339ce01-cdec-47ae-aac0-75ced2772655 cluster: cluster2 client: client.mirror 2024-04-24T00:40:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:40:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:40:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:40:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:40:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:40:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:40:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:40:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:40:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:40:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:40:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:40:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:40:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:40:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:40:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:40:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:40:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:40:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:40:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:40:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:40:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:40:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:40:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:40:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=9086], [object_number=9, tag_tid=2, entry_tid=9085], [object_number=8, tag_tid=2, entry_tid=9084], [object_number=11, tag_tid=2, entry_tid=9083]]], state=connected] 2024-04-24T00:40:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=9086], [object_number=9, tag_tid=2, entry_tid=9085], [object_number=8, tag_tid=2, entry_tid=9084], [object_number=11, tag_tid=2, entry_tid=9083]]], state=connected] 2024-04-24T00:40:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=9086' 2024-04-24T00:40:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=9086' 2024-04-24T00:40:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=9086' '!=' '' 2024-04-24T00:40:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=9086' = '' 2024-04-24T00:40:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=9086' 2024-04-24T00:40:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=9086' 2024-04-24T00:40:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=9086' 2024-04-24T00:40:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9086 2024-04-24T00:40:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 9086 2024-04-24T00:40:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=9203], [object_number=10, tag_tid=2, entry_tid=9202], [object_number=9, tag_tid=2, entry_tid=9201], [object_number=8, tag_tid=2, entry_tid=9200]]], state=connected] 2024-04-24T00:40:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=9203], [object_number=10, tag_tid=2, entry_tid=9202], [object_number=9, tag_tid=2, entry_tid=9201], [object_number=8, tag_tid=2, entry_tid=9200]]], state=connected] 2024-04-24T00:40:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=9203' 2024-04-24T00:40:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=9203' 2024-04-24T00:40:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=9203' '!=' 'object_number=10, tag_tid=2, entry_tid=9086' 2024-04-24T00:40:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=9203' = 'object_number=10, tag_tid=2, entry_tid=9086' 2024-04-24T00:40:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=9203' 2024-04-24T00:40:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=9203' 2024-04-24T00:40:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=9203' 2024-04-24T00:40:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9203 2024-04-24T00:40:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 9203 2024-04-24T00:40:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=9325], [object_number=8, tag_tid=2, entry_tid=9324], [object_number=11, tag_tid=2, entry_tid=9323], [object_number=10, tag_tid=2, entry_tid=9322]]], state=connected] 2024-04-24T00:40:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=9325], [object_number=8, tag_tid=2, entry_tid=9324], [object_number=11, tag_tid=2, entry_tid=9323], [object_number=10, tag_tid=2, entry_tid=9322]]], state=connected] 2024-04-24T00:40:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=9325' 2024-04-24T00:40:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=9325' 2024-04-24T00:40:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=9325' '!=' 'object_number=11, tag_tid=2, entry_tid=9203' 2024-04-24T00:40:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=9325' = 'object_number=11, tag_tid=2, entry_tid=9203' 2024-04-24T00:40:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=9325' 2024-04-24T00:40:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=9325' 2024-04-24T00:40:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=9325' 2024-04-24T00:40:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9325 2024-04-24T00:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 9325 2024-04-24T00:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=9434], [object_number=9, tag_tid=2, entry_tid=9433], [object_number=8, tag_tid=2, entry_tid=9432], [object_number=11, tag_tid=2, entry_tid=9431]]], state=connected] 2024-04-24T00:40:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=9434], [object_number=9, tag_tid=2, entry_tid=9433], [object_number=8, tag_tid=2, entry_tid=9432], [object_number=11, tag_tid=2, entry_tid=9431]]], state=connected] 2024-04-24T00:40:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=9434' 2024-04-24T00:40:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=9434' 2024-04-24T00:40:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=9434' '!=' 'object_number=9, tag_tid=2, entry_tid=9325' 2024-04-24T00:40:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=9434' = 'object_number=9, tag_tid=2, entry_tid=9325' 2024-04-24T00:40:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=9434' 2024-04-24T00:40:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=9434' 2024-04-24T00:40:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=9434' 2024-04-24T00:40:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9434 2024-04-24T00:40:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 9434 2024-04-24T00:40:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=9547], [object_number=10, tag_tid=2, entry_tid=9546], [object_number=9, tag_tid=2, entry_tid=9545], [object_number=8, tag_tid=2, entry_tid=9544]]], state=connected] 2024-04-24T00:40:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=9547], [object_number=10, tag_tid=2, entry_tid=9546], [object_number=9, tag_tid=2, entry_tid=9545], [object_number=8, tag_tid=2, entry_tid=9544]]], state=connected] 2024-04-24T00:40:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=9547' 2024-04-24T00:40:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=9547' 2024-04-24T00:40:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=9547' '!=' 'object_number=10, tag_tid=2, entry_tid=9434' 2024-04-24T00:40:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=9547' = 'object_number=10, tag_tid=2, entry_tid=9434' 2024-04-24T00:40:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=9547' 2024-04-24T00:40:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=9547' 2024-04-24T00:40:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=9547' 2024-04-24T00:40:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9547 2024-04-24T00:40:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 9547 2024-04-24T00:40:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=9654], [object_number=9, tag_tid=2, entry_tid=9653], [object_number=8, tag_tid=2, entry_tid=9652], [object_number=11, tag_tid=2, entry_tid=9651]]], state=connected] 2024-04-24T00:40:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=9654], [object_number=9, tag_tid=2, entry_tid=9653], [object_number=8, tag_tid=2, entry_tid=9652], [object_number=11, tag_tid=2, entry_tid=9651]]], state=connected] 2024-04-24T00:40:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=9654' 2024-04-24T00:40:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=9654' 2024-04-24T00:40:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=9654' '!=' 'object_number=11, tag_tid=2, entry_tid=9547' 2024-04-24T00:40:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=9654' = 'object_number=11, tag_tid=2, entry_tid=9547' 2024-04-24T00:40:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=9654' 2024-04-24T00:40:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=9654' 2024-04-24T00:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=9654' 2024-04-24T00:40:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9654 2024-04-24T00:40:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 9654 2024-04-24T00:40:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=9769], [object_number=8, tag_tid=2, entry_tid=9768], [object_number=11, tag_tid=2, entry_tid=9767], [object_number=10, tag_tid=2, entry_tid=9766]]], state=connected] 2024-04-24T00:40:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=9769], [object_number=8, tag_tid=2, entry_tid=9768], [object_number=11, tag_tid=2, entry_tid=9767], [object_number=10, tag_tid=2, entry_tid=9766]]], state=connected] 2024-04-24T00:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=9769' 2024-04-24T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=9769' 2024-04-24T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=9769' '!=' 'object_number=10, tag_tid=2, entry_tid=9654' 2024-04-24T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=9769' = 'object_number=10, tag_tid=2, entry_tid=9654' 2024-04-24T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=9769' 2024-04-24T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=9769' 2024-04-24T00:40:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=9769' 2024-04-24T00:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9769 2024-04-24T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 9769 2024-04-24T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=9888], [object_number=11, tag_tid=2, entry_tid=9887], [object_number=10, tag_tid=2, entry_tid=9886], [object_number=9, tag_tid=2, entry_tid=9885]]], state=connected] 2024-04-24T00:40:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=9888], [object_number=11, tag_tid=2, entry_tid=9887], [object_number=10, tag_tid=2, entry_tid=9886], [object_number=9, tag_tid=2, entry_tid=9885]]], state=connected] 2024-04-24T00:40:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=9888' 2024-04-24T00:40:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=9888' 2024-04-24T00:40:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=9888' '!=' 'object_number=9, tag_tid=2, entry_tid=9769' 2024-04-24T00:40:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=9888' = 'object_number=9, tag_tid=2, entry_tid=9769' 2024-04-24T00:40:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=9888' 2024-04-24T00:40:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=9888' 2024-04-24T00:40:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=9888' 2024-04-24T00:40:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9888 2024-04-24T00:40:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 9888 2024-04-24T00:40:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10005], [object_number=8, tag_tid=2, entry_tid=10004], [object_number=11, tag_tid=2, entry_tid=10003], [object_number=10, tag_tid=2, entry_tid=10002]]], state=connected] 2024-04-24T00:40:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10005], [object_number=8, tag_tid=2, entry_tid=10004], [object_number=11, tag_tid=2, entry_tid=10003], [object_number=10, tag_tid=2, entry_tid=10002]]], state=connected] 2024-04-24T00:40:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=10005' 2024-04-24T00:40:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=10005' 2024-04-24T00:40:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10005' '!=' 'object_number=8, tag_tid=2, entry_tid=9888' 2024-04-24T00:40:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10005' = 'object_number=8, tag_tid=2, entry_tid=9888' 2024-04-24T00:40:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=10005' 2024-04-24T00:40:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10005' 2024-04-24T00:40:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10005' 2024-04-24T00:40:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10005 2024-04-24T00:40:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 10005 2024-04-24T00:40:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=10128], [object_number=11, tag_tid=2, entry_tid=10127], [object_number=10, tag_tid=2, entry_tid=10126], [object_number=9, tag_tid=2, entry_tid=10125]]], state=connected] 2024-04-24T00:40:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=10128], [object_number=11, tag_tid=2, entry_tid=10127], [object_number=10, tag_tid=2, entry_tid=10126], [object_number=9, tag_tid=2, entry_tid=10125]]], state=connected] 2024-04-24T00:40:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=10128' 2024-04-24T00:40:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=10128' 2024-04-24T00:40:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=10128' '!=' 'object_number=9, tag_tid=2, entry_tid=10005' 2024-04-24T00:40:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=10128' = 'object_number=9, tag_tid=2, entry_tid=10005' 2024-04-24T00:40:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=10128' 2024-04-24T00:40:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=10128' 2024-04-24T00:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=10128' 2024-04-24T00:40:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10128 2024-04-24T00:40:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 10128 2024-04-24T00:40:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10233], [object_number=8, tag_tid=2, entry_tid=10232], [object_number=11, tag_tid=2, entry_tid=10231], [object_number=10, tag_tid=2, entry_tid=10230]]], state=connected] 2024-04-24T00:40:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10233], [object_number=8, tag_tid=2, entry_tid=10232], [object_number=11, tag_tid=2, entry_tid=10231], [object_number=10, tag_tid=2, entry_tid=10230]]], state=connected] 2024-04-24T00:40:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=10233' 2024-04-24T00:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=10233' 2024-04-24T00:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10233' '!=' 'object_number=8, tag_tid=2, entry_tid=10128' 2024-04-24T00:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10233' = 'object_number=8, tag_tid=2, entry_tid=10128' 2024-04-24T00:40:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=10233' 2024-04-24T00:40:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10233' 2024-04-24T00:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10233' 2024-04-24T00:40:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10233 2024-04-24T00:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 10233 2024-04-24T00:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=10339], [object_number=10, tag_tid=2, entry_tid=10338], [object_number=9, tag_tid=2, entry_tid=10337], [object_number=8, tag_tid=2, entry_tid=10336]]], state=connected] 2024-04-24T00:40:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=10339], [object_number=10, tag_tid=2, entry_tid=10338], [object_number=9, tag_tid=2, entry_tid=10337], [object_number=8, tag_tid=2, entry_tid=10336]]], state=connected] 2024-04-24T00:40:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=10339' 2024-04-24T00:40:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=10339' 2024-04-24T00:40:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=10339' '!=' 'object_number=9, tag_tid=2, entry_tid=10233' 2024-04-24T00:40:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=10339' = 'object_number=9, tag_tid=2, entry_tid=10233' 2024-04-24T00:40:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=10339' 2024-04-24T00:40:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=10339' 2024-04-24T00:40:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=10339' 2024-04-24T00:40:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10339 2024-04-24T00:40:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 10339 2024-04-24T00:40:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10450], [object_number=9, tag_tid=2, entry_tid=10449], [object_number=8, tag_tid=2, entry_tid=10448], [object_number=11, tag_tid=2, entry_tid=10447]]], state=connected] 2024-04-24T00:40:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10450], [object_number=9, tag_tid=2, entry_tid=10449], [object_number=8, tag_tid=2, entry_tid=10448], [object_number=11, tag_tid=2, entry_tid=10447]]], state=connected] 2024-04-24T00:40:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=10450' 2024-04-24T00:40:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=10450' 2024-04-24T00:40:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10450' '!=' 'object_number=11, tag_tid=2, entry_tid=10339' 2024-04-24T00:40:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10450' = 'object_number=11, tag_tid=2, entry_tid=10339' 2024-04-24T00:40:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=10450' 2024-04-24T00:40:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10450' 2024-04-24T00:40:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10450' 2024-04-24T00:40:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10450 2024-04-24T00:40:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 10450 2024-04-24T00:40:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=10564], [object_number=11, tag_tid=2, entry_tid=10563], [object_number=10, tag_tid=2, entry_tid=10562], [object_number=9, tag_tid=2, entry_tid=10561]]], state=connected] 2024-04-24T00:40:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=10564], [object_number=11, tag_tid=2, entry_tid=10563], [object_number=10, tag_tid=2, entry_tid=10562], [object_number=9, tag_tid=2, entry_tid=10561]]], state=connected] 2024-04-24T00:40:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=10564' 2024-04-24T00:40:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=10564' 2024-04-24T00:40:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=10564' '!=' 'object_number=10, tag_tid=2, entry_tid=10450' 2024-04-24T00:40:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=10564' = 'object_number=10, tag_tid=2, entry_tid=10450' 2024-04-24T00:40:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=10564' 2024-04-24T00:40:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=10564' 2024-04-24T00:40:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=10564' 2024-04-24T00:40:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10564 2024-04-24T00:40:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 10564 2024-04-24T00:40:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10682], [object_number=9, tag_tid=2, entry_tid=10681], [object_number=8, tag_tid=2, entry_tid=10680], [object_number=11, tag_tid=2, entry_tid=10679]]], state=connected] 2024-04-24T00:40:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10682], [object_number=9, tag_tid=2, entry_tid=10681], [object_number=8, tag_tid=2, entry_tid=10680], [object_number=11, tag_tid=2, entry_tid=10679]]], state=connected] 2024-04-24T00:40:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=10682' 2024-04-24T00:40:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=10682' 2024-04-24T00:40:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10682' '!=' 'object_number=8, tag_tid=2, entry_tid=10564' 2024-04-24T00:40:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10682' = 'object_number=8, tag_tid=2, entry_tid=10564' 2024-04-24T00:40:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=10682' 2024-04-24T00:40:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10682' 2024-04-24T00:40:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10682' 2024-04-24T00:40:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10682 2024-04-24T00:40:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 10682 2024-04-24T00:40:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=10807], [object_number=10, tag_tid=2, entry_tid=10806], [object_number=9, tag_tid=2, entry_tid=10805], [object_number=8, tag_tid=2, entry_tid=10804]]], state=connected] 2024-04-24T00:40:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=10807], [object_number=10, tag_tid=2, entry_tid=10806], [object_number=9, tag_tid=2, entry_tid=10805], [object_number=8, tag_tid=2, entry_tid=10804]]], state=connected] 2024-04-24T00:40:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=10807' 2024-04-24T00:40:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=10807' 2024-04-24T00:40:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=10807' '!=' 'object_number=10, tag_tid=2, entry_tid=10682' 2024-04-24T00:40:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=10807' = 'object_number=10, tag_tid=2, entry_tid=10682' 2024-04-24T00:40:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=10807' 2024-04-24T00:40:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=10807' 2024-04-24T00:40:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=10807' 2024-04-24T00:40:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10807 2024-04-24T00:40:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 10807 2024-04-24T00:40:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10925], [object_number=8, tag_tid=2, entry_tid=10924], [object_number=11, tag_tid=2, entry_tid=10923], [object_number=10, tag_tid=2, entry_tid=10922]]], state=connected] 2024-04-24T00:40:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10925], [object_number=8, tag_tid=2, entry_tid=10924], [object_number=11, tag_tid=2, entry_tid=10923], [object_number=10, tag_tid=2, entry_tid=10922]]], state=connected] 2024-04-24T00:40:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=10925' 2024-04-24T00:40:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=10925' 2024-04-24T00:40:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10925' '!=' 'object_number=11, tag_tid=2, entry_tid=10807' 2024-04-24T00:40:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10925' = 'object_number=11, tag_tid=2, entry_tid=10807' 2024-04-24T00:40:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=10925' 2024-04-24T00:40:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10925' 2024-04-24T00:40:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10925' 2024-04-24T00:40:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10925 2024-04-24T00:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 10925 2024-04-24T00:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=11042], [object_number=9, tag_tid=2, entry_tid=11041], [object_number=8, tag_tid=2, entry_tid=11040], [object_number=11, tag_tid=2, entry_tid=11039]]], state=connected] 2024-04-24T00:40:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=11042], [object_number=9, tag_tid=2, entry_tid=11041], [object_number=8, tag_tid=2, entry_tid=11040], [object_number=11, tag_tid=2, entry_tid=11039]]], state=connected] 2024-04-24T00:40:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=11042' 2024-04-24T00:40:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=11042' 2024-04-24T00:40:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=11042' '!=' 'object_number=9, tag_tid=2, entry_tid=10925' 2024-04-24T00:40:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=11042' = 'object_number=9, tag_tid=2, entry_tid=10925' 2024-04-24T00:40:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=11042' 2024-04-24T00:40:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=11042' 2024-04-24T00:40:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=11042' 2024-04-24T00:40:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11042 2024-04-24T00:40:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 11042 2024-04-24T00:40:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=11161], [object_number=8, tag_tid=2, entry_tid=11160], [object_number=11, tag_tid=2, entry_tid=11159], [object_number=10, tag_tid=2, entry_tid=11158]]], state=connected] 2024-04-24T00:40:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=11161], [object_number=8, tag_tid=2, entry_tid=11160], [object_number=11, tag_tid=2, entry_tid=11159], [object_number=10, tag_tid=2, entry_tid=11158]]], state=connected] 2024-04-24T00:40:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=11161' 2024-04-24T00:40:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=11161' 2024-04-24T00:40:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=11161' '!=' 'object_number=10, tag_tid=2, entry_tid=11042' 2024-04-24T00:40:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=11161' = 'object_number=10, tag_tid=2, entry_tid=11042' 2024-04-24T00:40:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=11161' 2024-04-24T00:40:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=11161' 2024-04-24T00:40:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=11161' 2024-04-24T00:40:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11161 2024-04-24T00:40:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 11161 2024-04-24T00:40:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=11273], [object_number=8, tag_tid=2, entry_tid=11272], [object_number=11, tag_tid=2, entry_tid=11271], [object_number=10, tag_tid=2, entry_tid=11270]]], state=connected] 2024-04-24T00:40:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=11273], [object_number=8, tag_tid=2, entry_tid=11272], [object_number=11, tag_tid=2, entry_tid=11271], [object_number=10, tag_tid=2, entry_tid=11270]]], state=connected] 2024-04-24T00:40:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=11273' 2024-04-24T00:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=11273' 2024-04-24T00:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=11273' '!=' 'object_number=9, tag_tid=2, entry_tid=11161' 2024-04-24T00:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=11273' = 'object_number=9, tag_tid=2, entry_tid=11161' 2024-04-24T00:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=11273' 2024-04-24T00:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=11273' 2024-04-24T00:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=11273' 2024-04-24T00:40:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11273 2024-04-24T00:40:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 11273 2024-04-24T00:40:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11395], [object_number=10, tag_tid=2, entry_tid=11394], [object_number=9, tag_tid=2, entry_tid=11393], [object_number=8, tag_tid=2, entry_tid=11392]]], state=connected] 2024-04-24T00:40:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11395], [object_number=10, tag_tid=2, entry_tid=11394], [object_number=9, tag_tid=2, entry_tid=11393], [object_number=8, tag_tid=2, entry_tid=11392]]], state=connected] 2024-04-24T00:40:26.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=11395' 2024-04-24T00:40:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=11395' 2024-04-24T00:40:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11395' '!=' 'object_number=9, tag_tid=2, entry_tid=11273' 2024-04-24T00:40:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11395' = 'object_number=9, tag_tid=2, entry_tid=11273' 2024-04-24T00:40:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=11395' 2024-04-24T00:40:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11395' 2024-04-24T00:40:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11395' 2024-04-24T00:40:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11395 2024-04-24T00:40:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 11395 2024-04-24T00:40:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11548], [object_number=11, tag_tid=2, entry_tid=11547], [object_number=10, tag_tid=2, entry_tid=11546], [object_number=9, tag_tid=2, entry_tid=11545]]], state=connected] 2024-04-24T00:40:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11548], [object_number=11, tag_tid=2, entry_tid=11547], [object_number=10, tag_tid=2, entry_tid=11546], [object_number=9, tag_tid=2, entry_tid=11545]]], state=connected] 2024-04-24T00:40:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=11548' 2024-04-24T00:40:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=11548' 2024-04-24T00:40:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11548' '!=' 'object_number=11, tag_tid=2, entry_tid=11395' 2024-04-24T00:40:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11548' = 'object_number=11, tag_tid=2, entry_tid=11395' 2024-04-24T00:40:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=11548' 2024-04-24T00:40:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11548' 2024-04-24T00:40:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11548' 2024-04-24T00:40:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11548 2024-04-24T00:40:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 11548 2024-04-24T00:40:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11703], [object_number=10, tag_tid=2, entry_tid=11702], [object_number=9, tag_tid=2, entry_tid=11701], [object_number=8, tag_tid=2, entry_tid=11700]]], state=connected] 2024-04-24T00:40:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11703], [object_number=10, tag_tid=2, entry_tid=11702], [object_number=9, tag_tid=2, entry_tid=11701], [object_number=8, tag_tid=2, entry_tid=11700]]], state=connected] 2024-04-24T00:40:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=11703' 2024-04-24T00:40:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=11703' 2024-04-24T00:40:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11703' '!=' 'object_number=8, tag_tid=2, entry_tid=11548' 2024-04-24T00:40:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11703' = 'object_number=8, tag_tid=2, entry_tid=11548' 2024-04-24T00:40:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=11703' 2024-04-24T00:40:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11703' 2024-04-24T00:40:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11703' 2024-04-24T00:40:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11703 2024-04-24T00:40:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 11703 2024-04-24T00:40:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:27.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:27.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:27.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11812], [object_number=11, tag_tid=2, entry_tid=11811], [object_number=10, tag_tid=2, entry_tid=11810], [object_number=9, tag_tid=2, entry_tid=11809]]], state=connected] 2024-04-24T00:40:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11812], [object_number=11, tag_tid=2, entry_tid=11811], [object_number=10, tag_tid=2, entry_tid=11810], [object_number=9, tag_tid=2, entry_tid=11809]]], state=connected] 2024-04-24T00:40:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=11812' 2024-04-24T00:40:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=11812' 2024-04-24T00:40:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11812' '!=' 'object_number=11, tag_tid=2, entry_tid=11703' 2024-04-24T00:40:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11812' = 'object_number=11, tag_tid=2, entry_tid=11703' 2024-04-24T00:40:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=11812' 2024-04-24T00:40:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11812' 2024-04-24T00:40:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11812' 2024-04-24T00:40:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11812 2024-04-24T00:40:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 11812 2024-04-24T00:40:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=11922], [object_number=9, tag_tid=2, entry_tid=11921], [object_number=8, tag_tid=2, entry_tid=11920], [object_number=11, tag_tid=2, entry_tid=11919]]], state=connected] 2024-04-24T00:40:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=11922], [object_number=9, tag_tid=2, entry_tid=11921], [object_number=8, tag_tid=2, entry_tid=11920], [object_number=11, tag_tid=2, entry_tid=11919]]], state=connected] 2024-04-24T00:40:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=11922' 2024-04-24T00:40:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=11922' 2024-04-24T00:40:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=11922' '!=' 'object_number=8, tag_tid=2, entry_tid=11812' 2024-04-24T00:40:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=11922' = 'object_number=8, tag_tid=2, entry_tid=11812' 2024-04-24T00:40:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=11922' 2024-04-24T00:40:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=11922' 2024-04-24T00:40:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=11922' 2024-04-24T00:40:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11922 2024-04-24T00:40:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 11922 2024-04-24T00:40:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=12042], [object_number=9, tag_tid=2, entry_tid=12041], [object_number=8, tag_tid=2, entry_tid=12040], [object_number=11, tag_tid=2, entry_tid=12039]]], state=connected] 2024-04-24T00:40:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=12042], [object_number=9, tag_tid=2, entry_tid=12041], [object_number=8, tag_tid=2, entry_tid=12040], [object_number=11, tag_tid=2, entry_tid=12039]]], state=connected] 2024-04-24T00:40:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=12042' 2024-04-24T00:40:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=12042' 2024-04-24T00:40:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=12042' '!=' 'object_number=10, tag_tid=2, entry_tid=11922' 2024-04-24T00:40:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=12042' = 'object_number=10, tag_tid=2, entry_tid=11922' 2024-04-24T00:40:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=12042' 2024-04-24T00:40:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=12042' 2024-04-24T00:40:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=12042' 2024-04-24T00:40:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12042 2024-04-24T00:40:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 12042 2024-04-24T00:40:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12160], [object_number=11, tag_tid=2, entry_tid=12159], [object_number=10, tag_tid=2, entry_tid=12158], [object_number=9, tag_tid=2, entry_tid=12157]]], state=connected] 2024-04-24T00:40:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12160], [object_number=11, tag_tid=2, entry_tid=12159], [object_number=10, tag_tid=2, entry_tid=12158], [object_number=9, tag_tid=2, entry_tid=12157]]], state=connected] 2024-04-24T00:40:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=12160' 2024-04-24T00:40:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=12160' 2024-04-24T00:40:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12160' '!=' 'object_number=10, tag_tid=2, entry_tid=12042' 2024-04-24T00:40:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12160' = 'object_number=10, tag_tid=2, entry_tid=12042' 2024-04-24T00:40:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=12160' 2024-04-24T00:40:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12160' 2024-04-24T00:40:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12160' 2024-04-24T00:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12160 2024-04-24T00:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 12160 2024-04-24T00:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=12327], [object_number=10, tag_tid=2, entry_tid=12326], [object_number=9, tag_tid=2, entry_tid=12325], [object_number=8, tag_tid=2, entry_tid=12324]]], state=connected] 2024-04-24T00:40:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=12327], [object_number=10, tag_tid=2, entry_tid=12326], [object_number=9, tag_tid=2, entry_tid=12325], [object_number=8, tag_tid=2, entry_tid=12324]]], state=connected] 2024-04-24T00:40:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=12327' 2024-04-24T00:40:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=12327' 2024-04-24T00:40:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=12327' '!=' 'object_number=8, tag_tid=2, entry_tid=12160' 2024-04-24T00:40:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=12327' = 'object_number=8, tag_tid=2, entry_tid=12160' 2024-04-24T00:40:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=12327' 2024-04-24T00:40:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=12327' 2024-04-24T00:40:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=12327' 2024-04-24T00:40:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12327 2024-04-24T00:40:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 12327 2024-04-24T00:40:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12440], [object_number=11, tag_tid=2, entry_tid=12439], [object_number=10, tag_tid=2, entry_tid=12438], [object_number=9, tag_tid=2, entry_tid=12437]]], state=connected] 2024-04-24T00:40:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12440], [object_number=11, tag_tid=2, entry_tid=12439], [object_number=10, tag_tid=2, entry_tid=12438], [object_number=9, tag_tid=2, entry_tid=12437]]], state=connected] 2024-04-24T00:40:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=12440' 2024-04-24T00:40:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=12440' 2024-04-24T00:40:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12440' '!=' 'object_number=11, tag_tid=2, entry_tid=12327' 2024-04-24T00:40:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12440' = 'object_number=11, tag_tid=2, entry_tid=12327' 2024-04-24T00:40:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=12440' 2024-04-24T00:40:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12440' 2024-04-24T00:40:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12440' 2024-04-24T00:40:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12440 2024-04-24T00:40:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 12440 2024-04-24T00:40:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12544], [object_number=11, tag_tid=2, entry_tid=12543], [object_number=10, tag_tid=2, entry_tid=12542], [object_number=9, tag_tid=2, entry_tid=12541]]], state=connected] 2024-04-24T00:40:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T00:40:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12544], [object_number=11, tag_tid=2, entry_tid=12543], [object_number=10, tag_tid=2, entry_tid=12542], [object_number=9, tag_tid=2, entry_tid=12541]]], state=connected] 2024-04-24T00:40:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=12544' 2024-04-24T00:40:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=12544' 2024-04-24T00:40:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12544' '!=' 'object_number=8, tag_tid=2, entry_tid=12440' 2024-04-24T00:40:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12544' = 'object_number=8, tag_tid=2, entry_tid=12440' 2024-04-24T00:40:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=12544' 2024-04-24T00:40:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12544' 2024-04-24T00:40:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12544' 2024-04-24T00:40:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12544 2024-04-24T00:40:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 12544 2024-04-24T00:40:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12614], [object_number=13, tag_tid=2, entry_tid=12613], [object_number=12, tag_tid=2, entry_tid=12612], [object_number=15, tag_tid=2, entry_tid=12611]]], state=connected] 2024-04-24T00:40:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12614], [object_number=13, tag_tid=2, entry_tid=12613], [object_number=12, tag_tid=2, entry_tid=12612], [object_number=15, tag_tid=2, entry_tid=12611]]], state=connected] 2024-04-24T00:40:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12614' 2024-04-24T00:40:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12614' 2024-04-24T00:40:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12614' '!=' 'object_number=8, tag_tid=2, entry_tid=12544' 2024-04-24T00:40:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12614' = 'object_number=8, tag_tid=2, entry_tid=12544' 2024-04-24T00:40:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12614' 2024-04-24T00:40:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12614' 2024-04-24T00:40:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12614' 2024-04-24T00:40:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12614 2024-04-24T00:40:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 12614 2024-04-24T00:40:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=12703], [object_number=14, tag_tid=2, entry_tid=12702], [object_number=13, tag_tid=2, entry_tid=12701], [object_number=12, tag_tid=2, entry_tid=12700]]], state=connected] 2024-04-24T00:40:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=12703], [object_number=14, tag_tid=2, entry_tid=12702], [object_number=13, tag_tid=2, entry_tid=12701], [object_number=12, tag_tid=2, entry_tid=12700]]], state=connected] 2024-04-24T00:40:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=12703' 2024-04-24T00:40:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=12703' 2024-04-24T00:40:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=12703' '!=' 'object_number=14, tag_tid=2, entry_tid=12614' 2024-04-24T00:40:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=12703' = 'object_number=14, tag_tid=2, entry_tid=12614' 2024-04-24T00:40:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=12703' 2024-04-24T00:40:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=12703' 2024-04-24T00:40:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=12703' 2024-04-24T00:40:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12703 2024-04-24T00:40:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 12703 2024-04-24T00:40:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12806], [object_number=13, tag_tid=2, entry_tid=12805], [object_number=12, tag_tid=2, entry_tid=12804], [object_number=15, tag_tid=2, entry_tid=12803]]], state=connected] 2024-04-24T00:40:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12806], [object_number=13, tag_tid=2, entry_tid=12805], [object_number=12, tag_tid=2, entry_tid=12804], [object_number=15, tag_tid=2, entry_tid=12803]]], state=connected] 2024-04-24T00:40:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12806' 2024-04-24T00:40:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12806' 2024-04-24T00:40:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12806' '!=' 'object_number=15, tag_tid=2, entry_tid=12703' 2024-04-24T00:40:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12806' = 'object_number=15, tag_tid=2, entry_tid=12703' 2024-04-24T00:40:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12806' 2024-04-24T00:40:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12806' 2024-04-24T00:40:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12806' 2024-04-24T00:40:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12806 2024-04-24T00:40:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 12806 2024-04-24T00:40:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=12908], [object_number=15, tag_tid=2, entry_tid=12907], [object_number=14, tag_tid=2, entry_tid=12906], [object_number=13, tag_tid=2, entry_tid=12905]]], state=connected] 2024-04-24T00:40:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.672 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T00:40:32.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T00:40:32.673 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T00:40:32.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T00:40:32.665+0000 7f2a6a91e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 29725) UID: 1000 2024-04-24T00:40:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=12908], [object_number=15, tag_tid=2, entry_tid=12907], [object_number=14, tag_tid=2, entry_tid=12906], [object_number=13, tag_tid=2, entry_tid=12905]]], state=connected] 2024-04-24T00:40:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=12908' 2024-04-24T00:40:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=12908' 2024-04-24T00:40:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=12908' '!=' 'object_number=14, tag_tid=2, entry_tid=12806' 2024-04-24T00:40:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=12908' = 'object_number=14, tag_tid=2, entry_tid=12806' 2024-04-24T00:40:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=12908' 2024-04-24T00:40:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=12908' 2024-04-24T00:40:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=12908' 2024-04-24T00:40:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12908 2024-04-24T00:40:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 12908 2024-04-24T00:40:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13012], [object_number=15, tag_tid=2, entry_tid=13011], [object_number=14, tag_tid=2, entry_tid=13010], [object_number=13, tag_tid=2, entry_tid=13009]]], state=connected] 2024-04-24T00:40:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13012], [object_number=15, tag_tid=2, entry_tid=13011], [object_number=14, tag_tid=2, entry_tid=13010], [object_number=13, tag_tid=2, entry_tid=13009]]], state=connected] 2024-04-24T00:40:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=13012' 2024-04-24T00:40:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=13012' 2024-04-24T00:40:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13012' '!=' 'object_number=12, tag_tid=2, entry_tid=12908' 2024-04-24T00:40:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13012' = 'object_number=12, tag_tid=2, entry_tid=12908' 2024-04-24T00:40:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=13012' 2024-04-24T00:40:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13012' 2024-04-24T00:40:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13012' 2024-04-24T00:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13012 2024-04-24T00:40:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13012 2024-04-24T00:40:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13115], [object_number=14, tag_tid=2, entry_tid=13114], [object_number=13, tag_tid=2, entry_tid=13113], [object_number=12, tag_tid=2, entry_tid=13112]]], state=connected] 2024-04-24T00:40:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13115], [object_number=14, tag_tid=2, entry_tid=13114], [object_number=13, tag_tid=2, entry_tid=13113], [object_number=12, tag_tid=2, entry_tid=13112]]], state=connected] 2024-04-24T00:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13115' 2024-04-24T00:40:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13115' 2024-04-24T00:40:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13115' '!=' 'object_number=12, tag_tid=2, entry_tid=13012' 2024-04-24T00:40:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13115' = 'object_number=12, tag_tid=2, entry_tid=13012' 2024-04-24T00:40:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13115' 2024-04-24T00:40:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13115' 2024-04-24T00:40:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13115' 2024-04-24T00:40:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13115 2024-04-24T00:40:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13115 2024-04-24T00:40:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13214], [object_number=13, tag_tid=2, entry_tid=13213], [object_number=12, tag_tid=2, entry_tid=13212], [object_number=15, tag_tid=2, entry_tid=13211]]], state=connected] 2024-04-24T00:40:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13214], [object_number=13, tag_tid=2, entry_tid=13213], [object_number=12, tag_tid=2, entry_tid=13212], [object_number=15, tag_tid=2, entry_tid=13211]]], state=connected] 2024-04-24T00:40:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=13214' 2024-04-24T00:40:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=13214' 2024-04-24T00:40:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13214' '!=' 'object_number=15, tag_tid=2, entry_tid=13115' 2024-04-24T00:40:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13214' = 'object_number=15, tag_tid=2, entry_tid=13115' 2024-04-24T00:40:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=13214' 2024-04-24T00:40:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13214' 2024-04-24T00:40:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13214' 2024-04-24T00:40:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13214 2024-04-24T00:40:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13214 2024-04-24T00:40:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13317], [object_number=12, tag_tid=2, entry_tid=13316], [object_number=15, tag_tid=2, entry_tid=13315], [object_number=14, tag_tid=2, entry_tid=13314]]], state=connected] 2024-04-24T00:40:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13317], [object_number=12, tag_tid=2, entry_tid=13316], [object_number=15, tag_tid=2, entry_tid=13315], [object_number=14, tag_tid=2, entry_tid=13314]]], state=connected] 2024-04-24T00:40:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=13317' 2024-04-24T00:40:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=13317' 2024-04-24T00:40:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13317' '!=' 'object_number=14, tag_tid=2, entry_tid=13214' 2024-04-24T00:40:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13317' = 'object_number=14, tag_tid=2, entry_tid=13214' 2024-04-24T00:40:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=13317' 2024-04-24T00:40:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13317' 2024-04-24T00:40:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13317' 2024-04-24T00:40:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13317 2024-04-24T00:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13317 2024-04-24T00:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13420], [object_number=15, tag_tid=2, entry_tid=13419], [object_number=14, tag_tid=2, entry_tid=13418], [object_number=13, tag_tid=2, entry_tid=13417]]], state=connected] 2024-04-24T00:40:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13420], [object_number=15, tag_tid=2, entry_tid=13419], [object_number=14, tag_tid=2, entry_tid=13418], [object_number=13, tag_tid=2, entry_tid=13417]]], state=connected] 2024-04-24T00:40:35.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=13420' 2024-04-24T00:40:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=13420' 2024-04-24T00:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13420' '!=' 'object_number=13, tag_tid=2, entry_tid=13317' 2024-04-24T00:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13420' = 'object_number=13, tag_tid=2, entry_tid=13317' 2024-04-24T00:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=13420' 2024-04-24T00:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13420' 2024-04-24T00:40:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13420' 2024-04-24T00:40:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13420 2024-04-24T00:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13420 2024-04-24T00:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13580], [object_number=15, tag_tid=2, entry_tid=13579], [object_number=14, tag_tid=2, entry_tid=13578], [object_number=13, tag_tid=2, entry_tid=13577]]], state=connected] 2024-04-24T00:40:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13580], [object_number=15, tag_tid=2, entry_tid=13579], [object_number=14, tag_tid=2, entry_tid=13578], [object_number=13, tag_tid=2, entry_tid=13577]]], state=connected] 2024-04-24T00:40:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=13580' 2024-04-24T00:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=13580' 2024-04-24T00:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13580' '!=' 'object_number=12, tag_tid=2, entry_tid=13420' 2024-04-24T00:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13580' = 'object_number=12, tag_tid=2, entry_tid=13420' 2024-04-24T00:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=13580' 2024-04-24T00:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13580' 2024-04-24T00:40:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13580' 2024-04-24T00:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13580 2024-04-24T00:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13580 2024-04-24T00:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13703], [object_number=14, tag_tid=2, entry_tid=13702], [object_number=13, tag_tid=2, entry_tid=13701], [object_number=12, tag_tid=2, entry_tid=13700]]], state=connected] 2024-04-24T00:40:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13703], [object_number=14, tag_tid=2, entry_tid=13702], [object_number=13, tag_tid=2, entry_tid=13701], [object_number=12, tag_tid=2, entry_tid=13700]]], state=connected] 2024-04-24T00:40:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13703' 2024-04-24T00:40:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13703' 2024-04-24T00:40:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13703' '!=' 'object_number=12, tag_tid=2, entry_tid=13580' 2024-04-24T00:40:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13703' = 'object_number=12, tag_tid=2, entry_tid=13580' 2024-04-24T00:40:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13703' 2024-04-24T00:40:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13703' 2024-04-24T00:40:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13703' 2024-04-24T00:40:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13703 2024-04-24T00:40:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13703 2024-04-24T00:40:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13798], [object_number=13, tag_tid=2, entry_tid=13797], [object_number=12, tag_tid=2, entry_tid=13796], [object_number=15, tag_tid=2, entry_tid=13795]]], state=connected] 2024-04-24T00:40:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13798], [object_number=13, tag_tid=2, entry_tid=13797], [object_number=12, tag_tid=2, entry_tid=13796], [object_number=15, tag_tid=2, entry_tid=13795]]], state=connected] 2024-04-24T00:40:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=13798' 2024-04-24T00:40:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=13798' 2024-04-24T00:40:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13798' '!=' 'object_number=15, tag_tid=2, entry_tid=13703' 2024-04-24T00:40:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13798' = 'object_number=15, tag_tid=2, entry_tid=13703' 2024-04-24T00:40:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=13798' 2024-04-24T00:40:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13798' 2024-04-24T00:40:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13798' 2024-04-24T00:40:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13798 2024-04-24T00:40:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13798 2024-04-24T00:40:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13894], [object_number=13, tag_tid=2, entry_tid=13893], [object_number=12, tag_tid=2, entry_tid=13892], [object_number=15, tag_tid=2, entry_tid=13891]]], state=connected] 2024-04-24T00:40:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13894], [object_number=13, tag_tid=2, entry_tid=13893], [object_number=12, tag_tid=2, entry_tid=13892], [object_number=15, tag_tid=2, entry_tid=13891]]], state=connected] 2024-04-24T00:40:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=13894' 2024-04-24T00:40:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=13894' 2024-04-24T00:40:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13894' '!=' 'object_number=14, tag_tid=2, entry_tid=13798' 2024-04-24T00:40:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13894' = 'object_number=14, tag_tid=2, entry_tid=13798' 2024-04-24T00:40:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=13894' 2024-04-24T00:40:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13894' 2024-04-24T00:40:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13894' 2024-04-24T00:40:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13894 2024-04-24T00:40:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13894 2024-04-24T00:40:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13994], [object_number=13, tag_tid=2, entry_tid=13993], [object_number=12, tag_tid=2, entry_tid=13992], [object_number=15, tag_tid=2, entry_tid=13991]]], state=connected] 2024-04-24T00:40:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13994], [object_number=13, tag_tid=2, entry_tid=13993], [object_number=12, tag_tid=2, entry_tid=13992], [object_number=15, tag_tid=2, entry_tid=13991]]], state=connected] 2024-04-24T00:40:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=13994' 2024-04-24T00:40:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=13994' 2024-04-24T00:40:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13994' '!=' 'object_number=14, tag_tid=2, entry_tid=13894' 2024-04-24T00:40:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13994' = 'object_number=14, tag_tid=2, entry_tid=13894' 2024-04-24T00:40:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=13994' 2024-04-24T00:40:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13994' 2024-04-24T00:40:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13994' 2024-04-24T00:40:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13994 2024-04-24T00:40:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 13994 2024-04-24T00:40:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14088], [object_number=15, tag_tid=2, entry_tid=14087], [object_number=14, tag_tid=2, entry_tid=14086], [object_number=13, tag_tid=2, entry_tid=14085]]], state=connected] 2024-04-24T00:40:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14088], [object_number=15, tag_tid=2, entry_tid=14087], [object_number=14, tag_tid=2, entry_tid=14086], [object_number=13, tag_tid=2, entry_tid=14085]]], state=connected] 2024-04-24T00:40:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=14088' 2024-04-24T00:40:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=14088' 2024-04-24T00:40:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14088' '!=' 'object_number=14, tag_tid=2, entry_tid=13994' 2024-04-24T00:40:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14088' = 'object_number=14, tag_tid=2, entry_tid=13994' 2024-04-24T00:40:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=14088' 2024-04-24T00:40:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14088' 2024-04-24T00:40:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14088' 2024-04-24T00:40:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14088 2024-04-24T00:40:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14088 2024-04-24T00:40:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14178], [object_number=13, tag_tid=2, entry_tid=14177], [object_number=12, tag_tid=2, entry_tid=14176], [object_number=15, tag_tid=2, entry_tid=14175]]], state=connected] 2024-04-24T00:40:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14178], [object_number=13, tag_tid=2, entry_tid=14177], [object_number=12, tag_tid=2, entry_tid=14176], [object_number=15, tag_tid=2, entry_tid=14175]]], state=connected] 2024-04-24T00:40:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=14178' 2024-04-24T00:40:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=14178' 2024-04-24T00:40:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14178' '!=' 'object_number=12, tag_tid=2, entry_tid=14088' 2024-04-24T00:40:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14178' = 'object_number=12, tag_tid=2, entry_tid=14088' 2024-04-24T00:40:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=14178' 2024-04-24T00:40:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14178' 2024-04-24T00:40:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14178' 2024-04-24T00:40:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14178 2024-04-24T00:40:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14178 2024-04-24T00:40:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:39.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=14271], [object_number=14, tag_tid=2, entry_tid=14270], [object_number=13, tag_tid=2, entry_tid=14269], [object_number=12, tag_tid=2, entry_tid=14268]]], state=connected] 2024-04-24T00:40:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=14271], [object_number=14, tag_tid=2, entry_tid=14270], [object_number=13, tag_tid=2, entry_tid=14269], [object_number=12, tag_tid=2, entry_tid=14268]]], state=connected] 2024-04-24T00:40:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=14271' 2024-04-24T00:40:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=14271' 2024-04-24T00:40:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=14271' '!=' 'object_number=14, tag_tid=2, entry_tid=14178' 2024-04-24T00:40:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=14271' = 'object_number=14, tag_tid=2, entry_tid=14178' 2024-04-24T00:40:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=14271' 2024-04-24T00:40:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=14271' 2024-04-24T00:40:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=14271' 2024-04-24T00:40:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14271 2024-04-24T00:40:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14271 2024-04-24T00:40:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14368], [object_number=15, tag_tid=2, entry_tid=14367], [object_number=14, tag_tid=2, entry_tid=14366], [object_number=13, tag_tid=2, entry_tid=14365]]], state=connected] 2024-04-24T00:40:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14368], [object_number=15, tag_tid=2, entry_tid=14367], [object_number=14, tag_tid=2, entry_tid=14366], [object_number=13, tag_tid=2, entry_tid=14365]]], state=connected] 2024-04-24T00:40:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=14368' 2024-04-24T00:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=14368' 2024-04-24T00:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14368' '!=' 'object_number=15, tag_tid=2, entry_tid=14271' 2024-04-24T00:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14368' = 'object_number=15, tag_tid=2, entry_tid=14271' 2024-04-24T00:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=14368' 2024-04-24T00:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14368' 2024-04-24T00:40:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14368' 2024-04-24T00:40:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14368 2024-04-24T00:40:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14368 2024-04-24T00:40:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=14469], [object_number=12, tag_tid=2, entry_tid=14468], [object_number=15, tag_tid=2, entry_tid=14467], [object_number=14, tag_tid=2, entry_tid=14466]]], state=connected] 2024-04-24T00:40:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=14469], [object_number=12, tag_tid=2, entry_tid=14468], [object_number=15, tag_tid=2, entry_tid=14467], [object_number=14, tag_tid=2, entry_tid=14466]]], state=connected] 2024-04-24T00:40:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=14469' 2024-04-24T00:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=14469' 2024-04-24T00:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=14469' '!=' 'object_number=12, tag_tid=2, entry_tid=14368' 2024-04-24T00:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=14469' = 'object_number=12, tag_tid=2, entry_tid=14368' 2024-04-24T00:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=14469' 2024-04-24T00:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=14469' 2024-04-24T00:40:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=14469' 2024-04-24T00:40:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14469 2024-04-24T00:40:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14469 2024-04-24T00:40:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14574], [object_number=13, tag_tid=2, entry_tid=14573], [object_number=12, tag_tid=2, entry_tid=14572], [object_number=15, tag_tid=2, entry_tid=14571]]], state=connected] 2024-04-24T00:40:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14574], [object_number=13, tag_tid=2, entry_tid=14573], [object_number=12, tag_tid=2, entry_tid=14572], [object_number=15, tag_tid=2, entry_tid=14571]]], state=connected] 2024-04-24T00:40:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=14574' 2024-04-24T00:40:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=14574' 2024-04-24T00:40:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14574' '!=' 'object_number=13, tag_tid=2, entry_tid=14469' 2024-04-24T00:40:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14574' = 'object_number=13, tag_tid=2, entry_tid=14469' 2024-04-24T00:40:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=14574' 2024-04-24T00:40:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14574' 2024-04-24T00:40:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14574' 2024-04-24T00:40:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14574 2024-04-24T00:40:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14574 2024-04-24T00:40:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:40.673 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T00:40:40.673 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:40:40.674 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T00:40:40.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T00:40:40.674 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T00:40:40.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T00:40:40.677 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2024-04-24T00:40:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14686], [object_number=13, tag_tid=2, entry_tid=14685], [object_number=12, tag_tid=2, entry_tid=14684], [object_number=15, tag_tid=2, entry_tid=14683]]], state=connected] 2024-04-24T00:40:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14686], [object_number=13, tag_tid=2, entry_tid=14685], [object_number=12, tag_tid=2, entry_tid=14684], [object_number=15, tag_tid=2, entry_tid=14683]]], state=connected] 2024-04-24T00:40:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=14686' 2024-04-24T00:40:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=14686' 2024-04-24T00:40:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14686' '!=' 'object_number=14, tag_tid=2, entry_tid=14574' 2024-04-24T00:40:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14686' = 'object_number=14, tag_tid=2, entry_tid=14574' 2024-04-24T00:40:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=14686' 2024-04-24T00:40:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14686' 2024-04-24T00:40:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14686' 2024-04-24T00:40:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14686 2024-04-24T00:40:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14686 2024-04-24T00:40:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14790], [object_number=13, tag_tid=2, entry_tid=14789], [object_number=12, tag_tid=2, entry_tid=14788], [object_number=15, tag_tid=2, entry_tid=14787]]], state=connected] 2024-04-24T00:40:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14790], [object_number=13, tag_tid=2, entry_tid=14789], [object_number=12, tag_tid=2, entry_tid=14788], [object_number=15, tag_tid=2, entry_tid=14787]]], state=connected] 2024-04-24T00:40:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=14790' 2024-04-24T00:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=14790' 2024-04-24T00:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14790' '!=' 'object_number=14, tag_tid=2, entry_tid=14686' 2024-04-24T00:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14790' = 'object_number=14, tag_tid=2, entry_tid=14686' 2024-04-24T00:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=14790' 2024-04-24T00:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14790' 2024-04-24T00:40:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14790' 2024-04-24T00:40:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14790 2024-04-24T00:40:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14790 2024-04-24T00:40:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=14881], [object_number=12, tag_tid=2, entry_tid=14880], [object_number=15, tag_tid=2, entry_tid=14879], [object_number=14, tag_tid=2, entry_tid=14878]]], state=connected] 2024-04-24T00:40:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=14881], [object_number=12, tag_tid=2, entry_tid=14880], [object_number=15, tag_tid=2, entry_tid=14879], [object_number=14, tag_tid=2, entry_tid=14878]]], state=connected] 2024-04-24T00:40:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=14881' 2024-04-24T00:40:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=14881' 2024-04-24T00:40:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=14881' '!=' 'object_number=14, tag_tid=2, entry_tid=14790' 2024-04-24T00:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=14881' = 'object_number=14, tag_tid=2, entry_tid=14790' 2024-04-24T00:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=14881' 2024-04-24T00:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=14881' 2024-04-24T00:40:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=14881' 2024-04-24T00:40:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14881 2024-04-24T00:40:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14881 2024-04-24T00:40:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14908], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906], [object_number=13, tag_tid=2, entry_tid=14905]]], state=connected] 2024-04-24T00:40:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14908], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906], [object_number=13, tag_tid=2, entry_tid=14905]]], state=connected] 2024-04-24T00:40:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=14908' 2024-04-24T00:40:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=14908' 2024-04-24T00:40:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14908' '!=' 'object_number=13, tag_tid=2, entry_tid=14881' 2024-04-24T00:40:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:40:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14908' = 'object_number=13, tag_tid=2, entry_tid=14881' 2024-04-24T00:40:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=14908' 2024-04-24T00:40:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T00:40:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14908' 2024-04-24T00:40:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:40:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T00:40:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:40:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14908' 2024-04-24T00:40:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:40:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:40:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14908 2024-04-24T00:40:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 14908 2024-04-24T00:40:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:40:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:40:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:40:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:40:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:40:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:40:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:40:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:40:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:40:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:40:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:40:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:40:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:40:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:40:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:40:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:40:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:40:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:40:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:40:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:40:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:40:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:40:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:40:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:40:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:40:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:40:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:40:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:40:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:40:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T00:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=14907], [object_number=14, tag_tid=2, entry_tid=14906]]], state=connected] 2024-04-24T00:40:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:40:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=3, entry_tid=1' 2024-04-24T00:40:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-04-24T00:40:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1 2024-04-24T00:40:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:40:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:40:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:40:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-04-24T00:40:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:40:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:40:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:40:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T00:40:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:40:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:40:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:40:29 2024 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:40:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:40:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap1 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:40:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:40:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:40:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:40:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:40:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:40:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:40:45.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:45.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap1-clean 2024-04-24T00:40:45.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-04-24T00:40:45.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:40:45.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1-clean 2024-04-24T00:40:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:40:45.671 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T00:40:45.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T00:40:45.671 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-24T00:40:45.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T00:40:45.669+0000 7fc704632640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 29647) UID: 1000 2024-04-24T00:40:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:40:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:40:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:40:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:40:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T00:40:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:40:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:40:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 34449408 bytes 2024-04-24T00:40:49.672 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T00:40:49.672 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:40:49.672 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T00:40:49.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T00:40:49.673 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T00:40:49.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T00:40:49.675 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-04-24T00:40:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 109908992 bytes 2024-04-24T00:40:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 136838144 bytes 2024-04-24T00:40:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:40:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:40:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:40:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:40:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:40:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:40:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:40:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:40:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:40:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:40:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:40:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:40:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:40:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:40:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:40:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:40:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:40:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:40:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T00:40:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T00:40:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24T00:40:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:40:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:40:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T00:40:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T00:40:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: ' 2024-04-24T00:40:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:40:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:40:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:40:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: ' 2024-04-24T00:40:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:40:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:40:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 4 2024-04-24T00:40:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:40:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:40:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:40:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:40:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:40:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:40:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:40:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:40:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:40:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: ' 2024-04-24T00:40:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:40:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:40:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:40:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T00:41:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:41:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:41:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:41:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:41:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:41:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:41:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:41:05' 2024-04-24T00:41:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:41:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:41:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T00:41:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:41:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:41:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:41:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:41:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:41:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:41:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:41:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:41:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:41:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:41:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":13,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":13,"tag_tid":4}} 2024-04-24T00:41:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T00:41:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:41:14 2024-04-24T00:41:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:41:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:41:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:41:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:41:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:41:05' 2024-04-24T00:41:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:41:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:41:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:41:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":13,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":13,"tag_tid":4}} 2024-04-24T00:41:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:41:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:41:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:41:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:41:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:41:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T00:41:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:41:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:41:14' 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:41:14' 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:41:14' 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:41:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:41:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142114271mirror.1smithi13419.0.0-2424-gb98aab6ctrueOK42194274mirror.3smithi13419.0.0-2424-gb98aab6cfalseOK42594312mirror.2smithi13419.0.0-2424-gb98aab6cfalseOK43304333mirror.0smithi13419.0.0-2424-gb98aab6cfalseOKtestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":13,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":13,"tag_tid":4}}42114271mirror.1smithi1342024-04-24 00:41:14cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:41:05' 2024-04-24T00:41:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:41:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:41:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:41:14' 2024-04-24T00:41:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:41:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:41:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:41:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:41:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:41:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2 2024-04-24T00:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap2 2024-04-24T00:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2 2024-04-24T00:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:41:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:41:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:41:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:41:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:41:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:41:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:41:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:41:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:41:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:41:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:41:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:41:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:41:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:41:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:41:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:41:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:41:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:41:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:41:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:41:24.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:41:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:41:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:41:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:41:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:41:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:41:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:41:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:41:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:41:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:41:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:41:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:41:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:41:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:41:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:41:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:41:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:41:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=5, entry_tid=1682], [object_number=13, tag_tid=5, entry_tid=1681], [object_number=12, tag_tid=5, entry_tid=1680], [object_number=15, tag_tid=5, entry_tid=1679]]], state=connected] 2024-04-24T00:41:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:41:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=5, entry_tid=1682], [object_number=13, tag_tid=5, entry_tid=1681], [object_number=12, tag_tid=5, entry_tid=1680], [object_number=15, tag_tid=5, entry_tid=1679]]], state=connected] 2024-04-24T00:41:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=5, entry_tid=1682' 2024-04-24T00:41:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=14, tag_tid=5, entry_tid=1682' 2024-04-24T00:41:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=5, entry_tid=1682' '!=' '' 2024-04-24T00:41:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=5, entry_tid=1682' = '' 2024-04-24T00:41:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=5, entry_tid=1682' 2024-04-24T00:41:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=5, entry_tid=1682' 2024-04-24T00:41:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=5, entry_tid=1682' 2024-04-24T00:41:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=1682 2024-04-24T00:41:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 1682 2024-04-24T00:41:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:41:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=1784], [object_number=15, tag_tid=5, entry_tid=1783], [object_number=18, tag_tid=5, entry_tid=1782], [object_number=13, tag_tid=5, entry_tid=1781]]], state=connected] 2024-04-24T00:41:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T00:41:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=1784], [object_number=15, tag_tid=5, entry_tid=1783], [object_number=18, tag_tid=5, entry_tid=1782], [object_number=13, tag_tid=5, entry_tid=1781]]], state=connected] 2024-04-24T00:41:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=16, tag_tid=5, entry_tid=1784' 2024-04-24T00:41:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=16, tag_tid=5, entry_tid=1784' 2024-04-24T00:41:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=1784' '!=' 'object_number=14, tag_tid=5, entry_tid=1682' 2024-04-24T00:41:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=1784' = 'object_number=14, tag_tid=5, entry_tid=1682' 2024-04-24T00:41:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=16, tag_tid=5, entry_tid=1784' 2024-04-24T00:41:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=1784' 2024-04-24T00:41:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=1784' 2024-04-24T00:41:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=1784 2024-04-24T00:41:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 1784 2024-04-24T00:41:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=1883], [object_number=18, tag_tid=5, entry_tid=1882], [object_number=17, tag_tid=5, entry_tid=1881], [object_number=16, tag_tid=5, entry_tid=1880]]], state=connected] 2024-04-24T00:41:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=1883], [object_number=18, tag_tid=5, entry_tid=1882], [object_number=17, tag_tid=5, entry_tid=1881], [object_number=16, tag_tid=5, entry_tid=1880]]], state=connected] 2024-04-24T00:41:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=19, tag_tid=5, entry_tid=1883' 2024-04-24T00:41:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=19, tag_tid=5, entry_tid=1883' 2024-04-24T00:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=1883' '!=' 'object_number=16, tag_tid=5, entry_tid=1784' 2024-04-24T00:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=1883' = 'object_number=16, tag_tid=5, entry_tid=1784' 2024-04-24T00:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=19, tag_tid=5, entry_tid=1883' 2024-04-24T00:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=1883' 2024-04-24T00:41:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=1883' 2024-04-24T00:41:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=1883 2024-04-24T00:41:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 1883 2024-04-24T00:41:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=2001], [object_number=16, tag_tid=5, entry_tid=2000], [object_number=19, tag_tid=5, entry_tid=1999], [object_number=18, tag_tid=5, entry_tid=1998]]], state=connected] 2024-04-24T00:41:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:27.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=2001], [object_number=16, tag_tid=5, entry_tid=2000], [object_number=19, tag_tid=5, entry_tid=1999], [object_number=18, tag_tid=5, entry_tid=1998]]], state=connected] 2024-04-24T00:41:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=17, tag_tid=5, entry_tid=2001' 2024-04-24T00:41:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=17, tag_tid=5, entry_tid=2001' 2024-04-24T00:41:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=2001' '!=' 'object_number=19, tag_tid=5, entry_tid=1883' 2024-04-24T00:41:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=2001' = 'object_number=19, tag_tid=5, entry_tid=1883' 2024-04-24T00:41:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=17, tag_tid=5, entry_tid=2001' 2024-04-24T00:41:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=2001' 2024-04-24T00:41:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=2001' 2024-04-24T00:41:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2001 2024-04-24T00:41:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 2001 2024-04-24T00:41:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=2118], [object_number=17, tag_tid=5, entry_tid=2117], [object_number=16, tag_tid=5, entry_tid=2116], [object_number=19, tag_tid=5, entry_tid=2115]]], state=connected] 2024-04-24T00:41:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=2118], [object_number=17, tag_tid=5, entry_tid=2117], [object_number=16, tag_tid=5, entry_tid=2116], [object_number=19, tag_tid=5, entry_tid=2115]]], state=connected] 2024-04-24T00:41:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=2118' 2024-04-24T00:41:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=2118' 2024-04-24T00:41:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=2118' '!=' 'object_number=17, tag_tid=5, entry_tid=2001' 2024-04-24T00:41:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=2118' = 'object_number=17, tag_tid=5, entry_tid=2001' 2024-04-24T00:41:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=2118' 2024-04-24T00:41:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=2118' 2024-04-24T00:41:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=2118' 2024-04-24T00:41:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2118 2024-04-24T00:41:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 2118 2024-04-24T00:41:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=2228], [object_number=19, tag_tid=5, entry_tid=2227], [object_number=18, tag_tid=5, entry_tid=2226], [object_number=17, tag_tid=5, entry_tid=2225]]], state=connected] 2024-04-24T00:41:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=2228], [object_number=19, tag_tid=5, entry_tid=2227], [object_number=18, tag_tid=5, entry_tid=2226], [object_number=17, tag_tid=5, entry_tid=2225]]], state=connected] 2024-04-24T00:41:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=16, tag_tid=5, entry_tid=2228' 2024-04-24T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=16, tag_tid=5, entry_tid=2228' 2024-04-24T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=2228' '!=' 'object_number=18, tag_tid=5, entry_tid=2118' 2024-04-24T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=2228' = 'object_number=18, tag_tid=5, entry_tid=2118' 2024-04-24T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=16, tag_tid=5, entry_tid=2228' 2024-04-24T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=2228' 2024-04-24T00:41:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=2228' 2024-04-24T00:41:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2228 2024-04-24T00:41:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 2228 2024-04-24T00:41:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=2359], [object_number=18, tag_tid=5, entry_tid=2358], [object_number=17, tag_tid=5, entry_tid=2357], [object_number=16, tag_tid=5, entry_tid=2356]]], state=connected] 2024-04-24T00:41:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=2359], [object_number=18, tag_tid=5, entry_tid=2358], [object_number=17, tag_tid=5, entry_tid=2357], [object_number=16, tag_tid=5, entry_tid=2356]]], state=connected] 2024-04-24T00:41:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=19, tag_tid=5, entry_tid=2359' 2024-04-24T00:41:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=19, tag_tid=5, entry_tid=2359' 2024-04-24T00:41:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=2359' '!=' 'object_number=16, tag_tid=5, entry_tid=2228' 2024-04-24T00:41:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=2359' = 'object_number=16, tag_tid=5, entry_tid=2228' 2024-04-24T00:41:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=19, tag_tid=5, entry_tid=2359' 2024-04-24T00:41:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=2359' 2024-04-24T00:41:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=2359' 2024-04-24T00:41:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2359 2024-04-24T00:41:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 2359 2024-04-24T00:41:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=2481], [object_number=16, tag_tid=5, entry_tid=2480], [object_number=19, tag_tid=5, entry_tid=2479], [object_number=18, tag_tid=5, entry_tid=2478]]], state=connected] 2024-04-24T00:41:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=2481], [object_number=16, tag_tid=5, entry_tid=2480], [object_number=19, tag_tid=5, entry_tid=2479], [object_number=18, tag_tid=5, entry_tid=2478]]], state=connected] 2024-04-24T00:41:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=17, tag_tid=5, entry_tid=2481' 2024-04-24T00:41:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=17, tag_tid=5, entry_tid=2481' 2024-04-24T00:41:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=2481' '!=' 'object_number=19, tag_tid=5, entry_tid=2359' 2024-04-24T00:41:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=2481' = 'object_number=19, tag_tid=5, entry_tid=2359' 2024-04-24T00:41:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=17, tag_tid=5, entry_tid=2481' 2024-04-24T00:41:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=2481' 2024-04-24T00:41:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=2481' 2024-04-24T00:41:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2481 2024-04-24T00:41:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 2481 2024-04-24T00:41:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:30.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=2623], [object_number=18, tag_tid=5, entry_tid=2622], [object_number=17, tag_tid=5, entry_tid=2621], [object_number=16, tag_tid=5, entry_tid=2620]]], state=connected] 2024-04-24T00:41:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=2623], [object_number=18, tag_tid=5, entry_tid=2622], [object_number=17, tag_tid=5, entry_tid=2621], [object_number=16, tag_tid=5, entry_tid=2620]]], state=connected] 2024-04-24T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=19, tag_tid=5, entry_tid=2623' 2024-04-24T00:41:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=19, tag_tid=5, entry_tid=2623' 2024-04-24T00:41:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=2623' '!=' 'object_number=17, tag_tid=5, entry_tid=2481' 2024-04-24T00:41:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=2623' = 'object_number=17, tag_tid=5, entry_tid=2481' 2024-04-24T00:41:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=19, tag_tid=5, entry_tid=2623' 2024-04-24T00:41:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=2623' 2024-04-24T00:41:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=2623' 2024-04-24T00:41:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2623 2024-04-24T00:41:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 2623 2024-04-24T00:41:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=2760], [object_number=19, tag_tid=5, entry_tid=2759], [object_number=18, tag_tid=5, entry_tid=2758], [object_number=17, tag_tid=5, entry_tid=2757]]], state=connected] 2024-04-24T00:41:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=2760], [object_number=19, tag_tid=5, entry_tid=2759], [object_number=18, tag_tid=5, entry_tid=2758], [object_number=17, tag_tid=5, entry_tid=2757]]], state=connected] 2024-04-24T00:41:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=16, tag_tid=5, entry_tid=2760' 2024-04-24T00:41:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=16, tag_tid=5, entry_tid=2760' 2024-04-24T00:41:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=2760' '!=' 'object_number=19, tag_tid=5, entry_tid=2623' 2024-04-24T00:41:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=2760' = 'object_number=19, tag_tid=5, entry_tid=2623' 2024-04-24T00:41:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=16, tag_tid=5, entry_tid=2760' 2024-04-24T00:41:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=2760' 2024-04-24T00:41:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=2760' 2024-04-24T00:41:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2760 2024-04-24T00:41:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 2760 2024-04-24T00:41:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=2882], [object_number=17, tag_tid=5, entry_tid=2881], [object_number=16, tag_tid=5, entry_tid=2880], [object_number=19, tag_tid=5, entry_tid=2879]]], state=connected] 2024-04-24T00:41:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=2882], [object_number=17, tag_tid=5, entry_tid=2881], [object_number=16, tag_tid=5, entry_tid=2880], [object_number=19, tag_tid=5, entry_tid=2879]]], state=connected] 2024-04-24T00:41:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=2882' 2024-04-24T00:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=2882' 2024-04-24T00:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=2882' '!=' 'object_number=16, tag_tid=5, entry_tid=2760' 2024-04-24T00:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=2882' = 'object_number=16, tag_tid=5, entry_tid=2760' 2024-04-24T00:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=2882' 2024-04-24T00:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=2882' 2024-04-24T00:41:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=2882' 2024-04-24T00:41:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2882 2024-04-24T00:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 2882 2024-04-24T00:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=2998], [object_number=17, tag_tid=5, entry_tid=2997], [object_number=16, tag_tid=5, entry_tid=2996], [object_number=19, tag_tid=5, entry_tid=2995]]], state=connected] 2024-04-24T00:41:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=2998], [object_number=17, tag_tid=5, entry_tid=2997], [object_number=16, tag_tid=5, entry_tid=2996], [object_number=19, tag_tid=5, entry_tid=2995]]], state=connected] 2024-04-24T00:41:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=2998' 2024-04-24T00:41:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=2998' 2024-04-24T00:41:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=2998' '!=' 'object_number=18, tag_tid=5, entry_tid=2882' 2024-04-24T00:41:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=2998' = 'object_number=18, tag_tid=5, entry_tid=2882' 2024-04-24T00:41:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=2998' 2024-04-24T00:41:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=2998' 2024-04-24T00:41:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=2998' 2024-04-24T00:41:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:32.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2998 2024-04-24T00:41:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 2998 2024-04-24T00:41:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=3120], [object_number=19, tag_tid=5, entry_tid=3119], [object_number=18, tag_tid=5, entry_tid=3118], [object_number=17, tag_tid=5, entry_tid=3117]]], state=connected] 2024-04-24T00:41:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=3120], [object_number=19, tag_tid=5, entry_tid=3119], [object_number=18, tag_tid=5, entry_tid=3118], [object_number=17, tag_tid=5, entry_tid=3117]]], state=connected] 2024-04-24T00:41:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=16, tag_tid=5, entry_tid=3120' 2024-04-24T00:41:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=16, tag_tid=5, entry_tid=3120' 2024-04-24T00:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=3120' '!=' 'object_number=18, tag_tid=5, entry_tid=2998' 2024-04-24T00:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=3120' = 'object_number=18, tag_tid=5, entry_tid=2998' 2024-04-24T00:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=16, tag_tid=5, entry_tid=3120' 2024-04-24T00:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=3120' 2024-04-24T00:41:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=3120' 2024-04-24T00:41:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3120 2024-04-24T00:41:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 3120 2024-04-24T00:41:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=3238], [object_number=17, tag_tid=5, entry_tid=3237], [object_number=16, tag_tid=5, entry_tid=3236], [object_number=19, tag_tid=5, entry_tid=3235]]], state=connected] 2024-04-24T00:41:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=3238], [object_number=17, tag_tid=5, entry_tid=3237], [object_number=16, tag_tid=5, entry_tid=3236], [object_number=19, tag_tid=5, entry_tid=3235]]], state=connected] 2024-04-24T00:41:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=3238' 2024-04-24T00:41:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=3238' 2024-04-24T00:41:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=3238' '!=' 'object_number=16, tag_tid=5, entry_tid=3120' 2024-04-24T00:41:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=3238' = 'object_number=16, tag_tid=5, entry_tid=3120' 2024-04-24T00:41:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=3238' 2024-04-24T00:41:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=3238' 2024-04-24T00:41:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=3238' 2024-04-24T00:41:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3238 2024-04-24T00:41:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 3238 2024-04-24T00:41:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=3362], [object_number=17, tag_tid=5, entry_tid=3361], [object_number=16, tag_tid=5, entry_tid=3360], [object_number=19, tag_tid=5, entry_tid=3359]]], state=connected] 2024-04-24T00:41:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=3362], [object_number=17, tag_tid=5, entry_tid=3361], [object_number=16, tag_tid=5, entry_tid=3360], [object_number=19, tag_tid=5, entry_tid=3359]]], state=connected] 2024-04-24T00:41:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=3362' 2024-04-24T00:41:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=3362' 2024-04-24T00:41:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=3362' '!=' 'object_number=18, tag_tid=5, entry_tid=3238' 2024-04-24T00:41:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=3362' = 'object_number=18, tag_tid=5, entry_tid=3238' 2024-04-24T00:41:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=3362' 2024-04-24T00:41:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=3362' 2024-04-24T00:41:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=3362' 2024-04-24T00:41:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3362 2024-04-24T00:41:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 3362 2024-04-24T00:41:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=3491], [object_number=18, tag_tid=5, entry_tid=3490], [object_number=17, tag_tid=5, entry_tid=3489], [object_number=16, tag_tid=5, entry_tid=3488]]], state=connected] 2024-04-24T00:41:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=3491], [object_number=18, tag_tid=5, entry_tid=3490], [object_number=17, tag_tid=5, entry_tid=3489], [object_number=16, tag_tid=5, entry_tid=3488]]], state=connected] 2024-04-24T00:41:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=19, tag_tid=5, entry_tid=3491' 2024-04-24T00:41:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=19, tag_tid=5, entry_tid=3491' 2024-04-24T00:41:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=3491' '!=' 'object_number=18, tag_tid=5, entry_tid=3362' 2024-04-24T00:41:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=3491' = 'object_number=18, tag_tid=5, entry_tid=3362' 2024-04-24T00:41:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=19, tag_tid=5, entry_tid=3491' 2024-04-24T00:41:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=3491' 2024-04-24T00:41:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=3491' 2024-04-24T00:41:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3491 2024-04-24T00:41:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 3491 2024-04-24T00:41:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=3608], [object_number=19, tag_tid=5, entry_tid=3607], [object_number=18, tag_tid=5, entry_tid=3606], [object_number=17, tag_tid=5, entry_tid=3605]]], state=connected] 2024-04-24T00:41:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=3608], [object_number=19, tag_tid=5, entry_tid=3607], [object_number=18, tag_tid=5, entry_tid=3606], [object_number=17, tag_tid=5, entry_tid=3605]]], state=connected] 2024-04-24T00:41:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=16, tag_tid=5, entry_tid=3608' 2024-04-24T00:41:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=16, tag_tid=5, entry_tid=3608' 2024-04-24T00:41:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=3608' '!=' 'object_number=19, tag_tid=5, entry_tid=3491' 2024-04-24T00:41:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=3608' = 'object_number=19, tag_tid=5, entry_tid=3491' 2024-04-24T00:41:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=16, tag_tid=5, entry_tid=3608' 2024-04-24T00:41:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=3608' 2024-04-24T00:41:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=3608' 2024-04-24T00:41:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3608 2024-04-24T00:41:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 3608 2024-04-24T00:41:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=3726], [object_number=17, tag_tid=5, entry_tid=3725], [object_number=16, tag_tid=5, entry_tid=3724], [object_number=19, tag_tid=5, entry_tid=3723]]], state=connected] 2024-04-24T00:41:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=3726], [object_number=17, tag_tid=5, entry_tid=3725], [object_number=16, tag_tid=5, entry_tid=3724], [object_number=19, tag_tid=5, entry_tid=3723]]], state=connected] 2024-04-24T00:41:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=3726' 2024-04-24T00:41:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=3726' 2024-04-24T00:41:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=3726' '!=' 'object_number=16, tag_tid=5, entry_tid=3608' 2024-04-24T00:41:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=3726' = 'object_number=16, tag_tid=5, entry_tid=3608' 2024-04-24T00:41:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=3726' 2024-04-24T00:41:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=3726' 2024-04-24T00:41:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=3726' 2024-04-24T00:41:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3726 2024-04-24T00:41:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 3726 2024-04-24T00:41:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=3848], [object_number=19, tag_tid=5, entry_tid=3847], [object_number=18, tag_tid=5, entry_tid=3846], [object_number=17, tag_tid=5, entry_tid=3845]]], state=connected] 2024-04-24T00:41:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=3848], [object_number=19, tag_tid=5, entry_tid=3847], [object_number=18, tag_tid=5, entry_tid=3846], [object_number=17, tag_tid=5, entry_tid=3845]]], state=connected] 2024-04-24T00:41:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=16, tag_tid=5, entry_tid=3848' 2024-04-24T00:41:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=16, tag_tid=5, entry_tid=3848' 2024-04-24T00:41:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=3848' '!=' 'object_number=18, tag_tid=5, entry_tid=3726' 2024-04-24T00:41:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=3848' = 'object_number=18, tag_tid=5, entry_tid=3726' 2024-04-24T00:41:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=16, tag_tid=5, entry_tid=3848' 2024-04-24T00:41:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=3848' 2024-04-24T00:41:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=3848' 2024-04-24T00:41:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3848 2024-04-24T00:41:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 3848 2024-04-24T00:41:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=3970], [object_number=17, tag_tid=5, entry_tid=3969], [object_number=16, tag_tid=5, entry_tid=3968], [object_number=19, tag_tid=5, entry_tid=3967]]], state=connected] 2024-04-24T00:41:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=3970], [object_number=17, tag_tid=5, entry_tid=3969], [object_number=16, tag_tid=5, entry_tid=3968], [object_number=19, tag_tid=5, entry_tid=3967]]], state=connected] 2024-04-24T00:41:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=3970' 2024-04-24T00:41:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=3970' 2024-04-24T00:41:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=3970' '!=' 'object_number=16, tag_tid=5, entry_tid=3848' 2024-04-24T00:41:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=3970' = 'object_number=16, tag_tid=5, entry_tid=3848' 2024-04-24T00:41:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=3970' 2024-04-24T00:41:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=3970' 2024-04-24T00:41:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=3970' 2024-04-24T00:41:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3970 2024-04-24T00:41:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 3970 2024-04-24T00:41:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=4095], [object_number=18, tag_tid=5, entry_tid=4094], [object_number=17, tag_tid=5, entry_tid=4093], [object_number=16, tag_tid=5, entry_tid=4092]]], state=connected] 2024-04-24T00:41:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=4095], [object_number=18, tag_tid=5, entry_tid=4094], [object_number=17, tag_tid=5, entry_tid=4093], [object_number=16, tag_tid=5, entry_tid=4092]]], state=connected] 2024-04-24T00:41:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=19, tag_tid=5, entry_tid=4095' 2024-04-24T00:41:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=19, tag_tid=5, entry_tid=4095' 2024-04-24T00:41:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=4095' '!=' 'object_number=18, tag_tid=5, entry_tid=3970' 2024-04-24T00:41:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=4095' = 'object_number=18, tag_tid=5, entry_tid=3970' 2024-04-24T00:41:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=19, tag_tid=5, entry_tid=4095' 2024-04-24T00:41:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=4095' 2024-04-24T00:41:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=4095' 2024-04-24T00:41:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=4095 2024-04-24T00:41:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 4095 2024-04-24T00:41:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=4222], [object_number=17, tag_tid=5, entry_tid=4221], [object_number=16, tag_tid=5, entry_tid=4220], [object_number=19, tag_tid=5, entry_tid=4219]]], state=connected] 2024-04-24T00:41:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=4222], [object_number=17, tag_tid=5, entry_tid=4221], [object_number=16, tag_tid=5, entry_tid=4220], [object_number=19, tag_tid=5, entry_tid=4219]]], state=connected] 2024-04-24T00:41:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=4222' 2024-04-24T00:41:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=4222' 2024-04-24T00:41:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=4222' '!=' 'object_number=19, tag_tid=5, entry_tid=4095' 2024-04-24T00:41:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=4222' = 'object_number=19, tag_tid=5, entry_tid=4095' 2024-04-24T00:41:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=4222' 2024-04-24T00:41:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=4222' 2024-04-24T00:41:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=4222' 2024-04-24T00:41:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=4222 2024-04-24T00:41:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 4222 2024-04-24T00:41:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=4360], [object_number=19, tag_tid=5, entry_tid=4359], [object_number=18, tag_tid=5, entry_tid=4358], [object_number=17, tag_tid=5, entry_tid=4357]]], state=connected] 2024-04-24T00:41:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=4360], [object_number=19, tag_tid=5, entry_tid=4359], [object_number=18, tag_tid=5, entry_tid=4358], [object_number=17, tag_tid=5, entry_tid=4357]]], state=connected] 2024-04-24T00:41:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=16, tag_tid=5, entry_tid=4360' 2024-04-24T00:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=16, tag_tid=5, entry_tid=4360' 2024-04-24T00:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=4360' '!=' 'object_number=18, tag_tid=5, entry_tid=4222' 2024-04-24T00:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=4360' = 'object_number=18, tag_tid=5, entry_tid=4222' 2024-04-24T00:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=16, tag_tid=5, entry_tid=4360' 2024-04-24T00:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=4360' 2024-04-24T00:41:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=4360' 2024-04-24T00:41:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=4360 2024-04-24T00:41:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 4360 2024-04-24T00:41:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=4505], [object_number=16, tag_tid=5, entry_tid=4504], [object_number=19, tag_tid=5, entry_tid=4503], [object_number=18, tag_tid=5, entry_tid=4502]]], state=connected] 2024-04-24T00:41:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=4505], [object_number=16, tag_tid=5, entry_tid=4504], [object_number=19, tag_tid=5, entry_tid=4503], [object_number=18, tag_tid=5, entry_tid=4502]]], state=connected] 2024-04-24T00:41:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=17, tag_tid=5, entry_tid=4505' 2024-04-24T00:41:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=17, tag_tid=5, entry_tid=4505' 2024-04-24T00:41:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=4505' '!=' 'object_number=16, tag_tid=5, entry_tid=4360' 2024-04-24T00:41:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=4505' = 'object_number=16, tag_tid=5, entry_tid=4360' 2024-04-24T00:41:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=17, tag_tid=5, entry_tid=4505' 2024-04-24T00:41:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=4505' 2024-04-24T00:41:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=4505' 2024-04-24T00:41:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=4505 2024-04-24T00:41:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 4505 2024-04-24T00:41:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=4636], [object_number=19, tag_tid=5, entry_tid=4635], [object_number=18, tag_tid=5, entry_tid=4634], [object_number=17, tag_tid=5, entry_tid=4633]]], state=connected] 2024-04-24T00:41:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=4636], [object_number=19, tag_tid=5, entry_tid=4635], [object_number=18, tag_tid=5, entry_tid=4634], [object_number=17, tag_tid=5, entry_tid=4633]]], state=connected] 2024-04-24T00:41:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=16, tag_tid=5, entry_tid=4636' 2024-04-24T00:41:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=16, tag_tid=5, entry_tid=4636' 2024-04-24T00:41:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=4636' '!=' 'object_number=17, tag_tid=5, entry_tid=4505' 2024-04-24T00:41:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=4636' = 'object_number=17, tag_tid=5, entry_tid=4505' 2024-04-24T00:41:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=16, tag_tid=5, entry_tid=4636' 2024-04-24T00:41:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=4636' 2024-04-24T00:41:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=4636' 2024-04-24T00:41:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=4636 2024-04-24T00:41:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 4636 2024-04-24T00:41:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=4752], [object_number=19, tag_tid=5, entry_tid=4751], [object_number=18, tag_tid=5, entry_tid=4750], [object_number=17, tag_tid=5, entry_tid=4749]]], state=connected] 2024-04-24T00:41:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=16, tag_tid=5, entry_tid=4752], [object_number=19, tag_tid=5, entry_tid=4751], [object_number=18, tag_tid=5, entry_tid=4750], [object_number=17, tag_tid=5, entry_tid=4749]]], state=connected] 2024-04-24T00:41:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=16, tag_tid=5, entry_tid=4752' 2024-04-24T00:41:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=16, tag_tid=5, entry_tid=4752' 2024-04-24T00:41:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=4752' '!=' 'object_number=16, tag_tid=5, entry_tid=4636' 2024-04-24T00:41:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=16, tag_tid=5, entry_tid=4752' = 'object_number=16, tag_tid=5, entry_tid=4636' 2024-04-24T00:41:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=16, tag_tid=5, entry_tid=4752' 2024-04-24T00:41:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=4752' 2024-04-24T00:41:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=16, tag_tid=5, entry_tid=4752' 2024-04-24T00:41:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=4752 2024-04-24T00:41:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 4752 2024-04-24T00:41:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=4874], [object_number=17, tag_tid=5, entry_tid=4873], [object_number=16, tag_tid=5, entry_tid=4872], [object_number=19, tag_tid=5, entry_tid=4871]]], state=connected] 2024-04-24T00:41:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=4874], [object_number=17, tag_tid=5, entry_tid=4873], [object_number=16, tag_tid=5, entry_tid=4872], [object_number=19, tag_tid=5, entry_tid=4871]]], state=connected] 2024-04-24T00:41:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=4874' 2024-04-24T00:41:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=4874' 2024-04-24T00:41:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=4874' '!=' 'object_number=16, tag_tid=5, entry_tid=4752' 2024-04-24T00:41:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=4874' = 'object_number=16, tag_tid=5, entry_tid=4752' 2024-04-24T00:41:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=4874' 2024-04-24T00:41:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=4874' 2024-04-24T00:41:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:41.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=4874' 2024-04-24T00:41:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=4874 2024-04-24T00:41:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 4874 2024-04-24T00:41:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=5001], [object_number=16, tag_tid=5, entry_tid=5000], [object_number=19, tag_tid=5, entry_tid=4999], [object_number=18, tag_tid=5, entry_tid=4998]]], state=connected] 2024-04-24T00:41:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=5001], [object_number=16, tag_tid=5, entry_tid=5000], [object_number=19, tag_tid=5, entry_tid=4999], [object_number=18, tag_tid=5, entry_tid=4998]]], state=connected] 2024-04-24T00:41:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=17, tag_tid=5, entry_tid=5001' 2024-04-24T00:41:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=17, tag_tid=5, entry_tid=5001' 2024-04-24T00:41:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=5001' '!=' 'object_number=18, tag_tid=5, entry_tid=4874' 2024-04-24T00:41:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=5001' = 'object_number=18, tag_tid=5, entry_tid=4874' 2024-04-24T00:41:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=17, tag_tid=5, entry_tid=5001' 2024-04-24T00:41:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=5001' 2024-04-24T00:41:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=5001' 2024-04-24T00:41:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5001 2024-04-24T00:41:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 5001 2024-04-24T00:41:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=5134], [object_number=17, tag_tid=5, entry_tid=5133], [object_number=16, tag_tid=5, entry_tid=5132], [object_number=19, tag_tid=5, entry_tid=5131]]], state=connected] 2024-04-24T00:41:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=5134], [object_number=17, tag_tid=5, entry_tid=5133], [object_number=16, tag_tid=5, entry_tid=5132], [object_number=19, tag_tid=5, entry_tid=5131]]], state=connected] 2024-04-24T00:41:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=5134' 2024-04-24T00:41:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=5134' 2024-04-24T00:41:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=5134' '!=' 'object_number=17, tag_tid=5, entry_tid=5001' 2024-04-24T00:41:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=5134' = 'object_number=17, tag_tid=5, entry_tid=5001' 2024-04-24T00:41:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=5134' 2024-04-24T00:41:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=5134' 2024-04-24T00:41:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=5134' 2024-04-24T00:41:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5134 2024-04-24T00:41:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 5134 2024-04-24T00:41:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=5269], [object_number=16, tag_tid=5, entry_tid=5268], [object_number=19, tag_tid=5, entry_tid=5267], [object_number=18, tag_tid=5, entry_tid=5266]]], state=connected] 2024-04-24T00:41:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=5269], [object_number=16, tag_tid=5, entry_tid=5268], [object_number=19, tag_tid=5, entry_tid=5267], [object_number=18, tag_tid=5, entry_tid=5266]]], state=connected] 2024-04-24T00:41:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=17, tag_tid=5, entry_tid=5269' 2024-04-24T00:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=17, tag_tid=5, entry_tid=5269' 2024-04-24T00:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=5269' '!=' 'object_number=18, tag_tid=5, entry_tid=5134' 2024-04-24T00:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=5269' = 'object_number=18, tag_tid=5, entry_tid=5134' 2024-04-24T00:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=17, tag_tid=5, entry_tid=5269' 2024-04-24T00:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=5269' 2024-04-24T00:41:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=5269' 2024-04-24T00:41:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5269 2024-04-24T00:41:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 5269 2024-04-24T00:41:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=5393], [object_number=16, tag_tid=5, entry_tid=5392], [object_number=19, tag_tid=5, entry_tid=5391], [object_number=18, tag_tid=5, entry_tid=5390]]], state=connected] 2024-04-24T00:41:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=5393], [object_number=16, tag_tid=5, entry_tid=5392], [object_number=19, tag_tid=5, entry_tid=5391], [object_number=18, tag_tid=5, entry_tid=5390]]], state=connected] 2024-04-24T00:41:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=17, tag_tid=5, entry_tid=5393' 2024-04-24T00:41:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=17, tag_tid=5, entry_tid=5393' 2024-04-24T00:41:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=5393' '!=' 'object_number=17, tag_tid=5, entry_tid=5269' 2024-04-24T00:41:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=5393' = 'object_number=17, tag_tid=5, entry_tid=5269' 2024-04-24T00:41:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=17, tag_tid=5, entry_tid=5393' 2024-04-24T00:41:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=5393' 2024-04-24T00:41:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=5393' 2024-04-24T00:41:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5393 2024-04-24T00:41:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 5393 2024-04-24T00:41:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=5530], [object_number=17, tag_tid=5, entry_tid=5529], [object_number=16, tag_tid=5, entry_tid=5528], [object_number=19, tag_tid=5, entry_tid=5527]]], state=connected] 2024-04-24T00:41:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=18, tag_tid=5, entry_tid=5530], [object_number=17, tag_tid=5, entry_tid=5529], [object_number=16, tag_tid=5, entry_tid=5528], [object_number=19, tag_tid=5, entry_tid=5527]]], state=connected] 2024-04-24T00:41:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=18, tag_tid=5, entry_tid=5530' 2024-04-24T00:41:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=18, tag_tid=5, entry_tid=5530' 2024-04-24T00:41:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=5530' '!=' 'object_number=17, tag_tid=5, entry_tid=5393' 2024-04-24T00:41:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=18, tag_tid=5, entry_tid=5530' = 'object_number=17, tag_tid=5, entry_tid=5393' 2024-04-24T00:41:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=18, tag_tid=5, entry_tid=5530' 2024-04-24T00:41:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=5530' 2024-04-24T00:41:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=18, tag_tid=5, entry_tid=5530' 2024-04-24T00:41:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5530 2024-04-24T00:41:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 5530 2024-04-24T00:41:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=5717], [object_number=16, tag_tid=5, entry_tid=5716], [object_number=19, tag_tid=5, entry_tid=5715], [object_number=18, tag_tid=5, entry_tid=5714]]], state=connected] 2024-04-24T00:41:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=17, tag_tid=5, entry_tid=5717], [object_number=16, tag_tid=5, entry_tid=5716], [object_number=19, tag_tid=5, entry_tid=5715], [object_number=18, tag_tid=5, entry_tid=5714]]], state=connected] 2024-04-24T00:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=17, tag_tid=5, entry_tid=5717' 2024-04-24T00:41:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=17, tag_tid=5, entry_tid=5717' 2024-04-24T00:41:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=5717' '!=' 'object_number=18, tag_tid=5, entry_tid=5530' 2024-04-24T00:41:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=17, tag_tid=5, entry_tid=5717' = 'object_number=18, tag_tid=5, entry_tid=5530' 2024-04-24T00:41:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=17, tag_tid=5, entry_tid=5717' 2024-04-24T00:41:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=5717' 2024-04-24T00:41:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=17, tag_tid=5, entry_tid=5717' 2024-04-24T00:41:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5717 2024-04-24T00:41:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 5717 2024-04-24T00:41:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=5863], [object_number=18, tag_tid=5, entry_tid=5862], [object_number=17, tag_tid=5, entry_tid=5861], [object_number=16, tag_tid=5, entry_tid=5860]]], state=connected] 2024-04-24T00:41:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 4 2024-04-24T00:41:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=19, tag_tid=5, entry_tid=5863], [object_number=18, tag_tid=5, entry_tid=5862], [object_number=17, tag_tid=5, entry_tid=5861], [object_number=16, tag_tid=5, entry_tid=5860]]], state=connected] 2024-04-24T00:41:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=19, tag_tid=5, entry_tid=5863' 2024-04-24T00:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=19, tag_tid=5, entry_tid=5863' 2024-04-24T00:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=5863' '!=' 'object_number=17, tag_tid=5, entry_tid=5717' 2024-04-24T00:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=19, tag_tid=5, entry_tid=5863' = 'object_number=17, tag_tid=5, entry_tid=5717' 2024-04-24T00:41:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=19, tag_tid=5, entry_tid=5863' 2024-04-24T00:41:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=5863' 2024-04-24T00:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=19, tag_tid=5, entry_tid=5863' 2024-04-24T00:41:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5863 2024-04-24T00:41:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 5863 2024-04-24T00:41:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=22, tag_tid=5, entry_tid=5982], [object_number=21, tag_tid=5, entry_tid=5981], [object_number=20, tag_tid=5, entry_tid=5980], [object_number=23, tag_tid=5, entry_tid=5979]]], state=connected] 2024-04-24T00:41:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=22, tag_tid=5, entry_tid=5982], [object_number=21, tag_tid=5, entry_tid=5981], [object_number=20, tag_tid=5, entry_tid=5980], [object_number=23, tag_tid=5, entry_tid=5979]]], state=connected] 2024-04-24T00:41:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=22, tag_tid=5, entry_tid=5982' 2024-04-24T00:41:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=22, tag_tid=5, entry_tid=5982' 2024-04-24T00:41:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=22, tag_tid=5, entry_tid=5982' '!=' 'object_number=19, tag_tid=5, entry_tid=5863' 2024-04-24T00:41:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=22, tag_tid=5, entry_tid=5982' = 'object_number=19, tag_tid=5, entry_tid=5863' 2024-04-24T00:41:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=22, tag_tid=5, entry_tid=5982' 2024-04-24T00:41:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=22, tag_tid=5, entry_tid=5982' 2024-04-24T00:41:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=22, tag_tid=5, entry_tid=5982' 2024-04-24T00:41:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5982 2024-04-24T00:41:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 5982 2024-04-24T00:41:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=22, tag_tid=5, entry_tid=6130], [object_number=21, tag_tid=5, entry_tid=6129], [object_number=20, tag_tid=5, entry_tid=6128], [object_number=23, tag_tid=5, entry_tid=6127]]], state=connected] 2024-04-24T00:41:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=22, tag_tid=5, entry_tid=6130], [object_number=21, tag_tid=5, entry_tid=6129], [object_number=20, tag_tid=5, entry_tid=6128], [object_number=23, tag_tid=5, entry_tid=6127]]], state=connected] 2024-04-24T00:41:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=22, tag_tid=5, entry_tid=6130' 2024-04-24T00:41:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=22, tag_tid=5, entry_tid=6130' 2024-04-24T00:41:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=22, tag_tid=5, entry_tid=6130' '!=' 'object_number=22, tag_tid=5, entry_tid=5982' 2024-04-24T00:41:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=22, tag_tid=5, entry_tid=6130' = 'object_number=22, tag_tid=5, entry_tid=5982' 2024-04-24T00:41:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=22, tag_tid=5, entry_tid=6130' 2024-04-24T00:41:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=22, tag_tid=5, entry_tid=6130' 2024-04-24T00:41:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=22, tag_tid=5, entry_tid=6130' 2024-04-24T00:41:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6130 2024-04-24T00:41:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 6130 2024-04-24T00:41:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=21, tag_tid=5, entry_tid=6269], [object_number=20, tag_tid=5, entry_tid=6268], [object_number=23, tag_tid=5, entry_tid=6267], [object_number=22, tag_tid=5, entry_tid=6266]]], state=connected] 2024-04-24T00:41:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=21, tag_tid=5, entry_tid=6269], [object_number=20, tag_tid=5, entry_tid=6268], [object_number=23, tag_tid=5, entry_tid=6267], [object_number=22, tag_tid=5, entry_tid=6266]]], state=connected] 2024-04-24T00:41:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=21, tag_tid=5, entry_tid=6269' 2024-04-24T00:41:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=21, tag_tid=5, entry_tid=6269' 2024-04-24T00:41:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=21, tag_tid=5, entry_tid=6269' '!=' 'object_number=22, tag_tid=5, entry_tid=6130' 2024-04-24T00:41:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=21, tag_tid=5, entry_tid=6269' = 'object_number=22, tag_tid=5, entry_tid=6130' 2024-04-24T00:41:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=21, tag_tid=5, entry_tid=6269' 2024-04-24T00:41:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=5, entry_tid=6269' 2024-04-24T00:41:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=5, entry_tid=6269' 2024-04-24T00:41:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6269 2024-04-24T00:41:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 6269 2024-04-24T00:41:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=6412], [object_number=23, tag_tid=5, entry_tid=6411], [object_number=22, tag_tid=5, entry_tid=6410], [object_number=21, tag_tid=5, entry_tid=6409]]], state=connected] 2024-04-24T00:41:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=6412], [object_number=23, tag_tid=5, entry_tid=6411], [object_number=22, tag_tid=5, entry_tid=6410], [object_number=21, tag_tid=5, entry_tid=6409]]], state=connected] 2024-04-24T00:41:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=20, tag_tid=5, entry_tid=6412' 2024-04-24T00:41:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=20, tag_tid=5, entry_tid=6412' 2024-04-24T00:41:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=6412' '!=' 'object_number=21, tag_tid=5, entry_tid=6269' 2024-04-24T00:41:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=6412' = 'object_number=21, tag_tid=5, entry_tid=6269' 2024-04-24T00:41:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=20, tag_tid=5, entry_tid=6412' 2024-04-24T00:41:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=6412' 2024-04-24T00:41:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=6412' 2024-04-24T00:41:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6412 2024-04-24T00:41:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 6412 2024-04-24T00:41:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=6532], [object_number=23, tag_tid=5, entry_tid=6531], [object_number=22, tag_tid=5, entry_tid=6530], [object_number=21, tag_tid=5, entry_tid=6529]]], state=connected] 2024-04-24T00:41:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=6532], [object_number=23, tag_tid=5, entry_tid=6531], [object_number=22, tag_tid=5, entry_tid=6530], [object_number=21, tag_tid=5, entry_tid=6529]]], state=connected] 2024-04-24T00:41:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=20, tag_tid=5, entry_tid=6532' 2024-04-24T00:41:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=20, tag_tid=5, entry_tid=6532' 2024-04-24T00:41:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=6532' '!=' 'object_number=20, tag_tid=5, entry_tid=6412' 2024-04-24T00:41:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=6532' = 'object_number=20, tag_tid=5, entry_tid=6412' 2024-04-24T00:41:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=20, tag_tid=5, entry_tid=6532' 2024-04-24T00:41:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=6532' 2024-04-24T00:41:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=6532' 2024-04-24T00:41:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6532 2024-04-24T00:41:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 6532 2024-04-24T00:41:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=6668], [object_number=23, tag_tid=5, entry_tid=6667], [object_number=22, tag_tid=5, entry_tid=6666], [object_number=21, tag_tid=5, entry_tid=6665]]], state=connected] 2024-04-24T00:41:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=6668], [object_number=23, tag_tid=5, entry_tid=6667], [object_number=22, tag_tid=5, entry_tid=6666], [object_number=21, tag_tid=5, entry_tid=6665]]], state=connected] 2024-04-24T00:41:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=20, tag_tid=5, entry_tid=6668' 2024-04-24T00:41:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=20, tag_tid=5, entry_tid=6668' 2024-04-24T00:41:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=6668' '!=' 'object_number=20, tag_tid=5, entry_tid=6532' 2024-04-24T00:41:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=6668' = 'object_number=20, tag_tid=5, entry_tid=6532' 2024-04-24T00:41:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=20, tag_tid=5, entry_tid=6668' 2024-04-24T00:41:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=6668' 2024-04-24T00:41:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=6668' 2024-04-24T00:41:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6668 2024-04-24T00:41:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 6668 2024-04-24T00:41:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=21, tag_tid=5, entry_tid=6805], [object_number=20, tag_tid=5, entry_tid=6804], [object_number=23, tag_tid=5, entry_tid=6803], [object_number=22, tag_tid=5, entry_tid=6802]]], state=connected] 2024-04-24T00:41:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=21, tag_tid=5, entry_tid=6805], [object_number=20, tag_tid=5, entry_tid=6804], [object_number=23, tag_tid=5, entry_tid=6803], [object_number=22, tag_tid=5, entry_tid=6802]]], state=connected] 2024-04-24T00:41:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=21, tag_tid=5, entry_tid=6805' 2024-04-24T00:41:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=21, tag_tid=5, entry_tid=6805' 2024-04-24T00:41:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=21, tag_tid=5, entry_tid=6805' '!=' 'object_number=20, tag_tid=5, entry_tid=6668' 2024-04-24T00:41:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=21, tag_tid=5, entry_tid=6805' = 'object_number=20, tag_tid=5, entry_tid=6668' 2024-04-24T00:41:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=21, tag_tid=5, entry_tid=6805' 2024-04-24T00:41:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=5, entry_tid=6805' 2024-04-24T00:41:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=5, entry_tid=6805' 2024-04-24T00:41:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6805 2024-04-24T00:41:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 6805 2024-04-24T00:41:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=21, tag_tid=5, entry_tid=6953], [object_number=20, tag_tid=5, entry_tid=6952], [object_number=23, tag_tid=5, entry_tid=6951], [object_number=22, tag_tid=5, entry_tid=6950]]], state=connected] 2024-04-24T00:41:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=21, tag_tid=5, entry_tid=6953], [object_number=20, tag_tid=5, entry_tid=6952], [object_number=23, tag_tid=5, entry_tid=6951], [object_number=22, tag_tid=5, entry_tid=6950]]], state=connected] 2024-04-24T00:41:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=21, tag_tid=5, entry_tid=6953' 2024-04-24T00:41:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=21, tag_tid=5, entry_tid=6953' 2024-04-24T00:41:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=21, tag_tid=5, entry_tid=6953' '!=' 'object_number=21, tag_tid=5, entry_tid=6805' 2024-04-24T00:41:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=21, tag_tid=5, entry_tid=6953' = 'object_number=21, tag_tid=5, entry_tid=6805' 2024-04-24T00:41:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=21, tag_tid=5, entry_tid=6953' 2024-04-24T00:41:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=5, entry_tid=6953' 2024-04-24T00:41:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=5, entry_tid=6953' 2024-04-24T00:41:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6953 2024-04-24T00:41:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 6953 2024-04-24T00:41:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=7132], [object_number=23, tag_tid=5, entry_tid=7131], [object_number=22, tag_tid=5, entry_tid=7130], [object_number=21, tag_tid=5, entry_tid=7129]]], state=connected] 2024-04-24T00:41:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=7132], [object_number=23, tag_tid=5, entry_tid=7131], [object_number=22, tag_tid=5, entry_tid=7130], [object_number=21, tag_tid=5, entry_tid=7129]]], state=connected] 2024-04-24T00:41:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=20, tag_tid=5, entry_tid=7132' 2024-04-24T00:41:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=20, tag_tid=5, entry_tid=7132' 2024-04-24T00:41:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=7132' '!=' 'object_number=21, tag_tid=5, entry_tid=6953' 2024-04-24T00:41:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=7132' = 'object_number=21, tag_tid=5, entry_tid=6953' 2024-04-24T00:41:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=20, tag_tid=5, entry_tid=7132' 2024-04-24T00:41:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=7132' 2024-04-24T00:41:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=7132' 2024-04-24T00:41:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7132 2024-04-24T00:41:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 7132 2024-04-24T00:41:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=7268], [object_number=23, tag_tid=5, entry_tid=7267], [object_number=22, tag_tid=5, entry_tid=7266], [object_number=21, tag_tid=5, entry_tid=7265]]], state=connected] 2024-04-24T00:41:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=7268], [object_number=23, tag_tid=5, entry_tid=7267], [object_number=22, tag_tid=5, entry_tid=7266], [object_number=21, tag_tid=5, entry_tid=7265]]], state=connected] 2024-04-24T00:41:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=20, tag_tid=5, entry_tid=7268' 2024-04-24T00:41:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=20, tag_tid=5, entry_tid=7268' 2024-04-24T00:41:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=7268' '!=' 'object_number=20, tag_tid=5, entry_tid=7132' 2024-04-24T00:41:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=7268' = 'object_number=20, tag_tid=5, entry_tid=7132' 2024-04-24T00:41:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=20, tag_tid=5, entry_tid=7268' 2024-04-24T00:41:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=7268' 2024-04-24T00:41:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=7268' 2024-04-24T00:41:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7268 2024-04-24T00:41:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 7268 2024-04-24T00:41:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=23, tag_tid=5, entry_tid=7391], [object_number=22, tag_tid=5, entry_tid=7390], [object_number=21, tag_tid=5, entry_tid=7389], [object_number=20, tag_tid=5, entry_tid=7388]]], state=connected] 2024-04-24T00:41:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=23, tag_tid=5, entry_tid=7391], [object_number=22, tag_tid=5, entry_tid=7390], [object_number=21, tag_tid=5, entry_tid=7389], [object_number=20, tag_tid=5, entry_tid=7388]]], state=connected] 2024-04-24T00:41:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=23, tag_tid=5, entry_tid=7391' 2024-04-24T00:41:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=23, tag_tid=5, entry_tid=7391' 2024-04-24T00:41:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=23, tag_tid=5, entry_tid=7391' '!=' 'object_number=20, tag_tid=5, entry_tid=7268' 2024-04-24T00:41:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=23, tag_tid=5, entry_tid=7391' = 'object_number=20, tag_tid=5, entry_tid=7268' 2024-04-24T00:41:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=23, tag_tid=5, entry_tid=7391' 2024-04-24T00:41:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=23, tag_tid=5, entry_tid=7391' 2024-04-24T00:41:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=23, tag_tid=5, entry_tid=7391' 2024-04-24T00:41:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7391 2024-04-24T00:41:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 7391 2024-04-24T00:41:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=7552], [object_number=23, tag_tid=5, entry_tid=7551], [object_number=22, tag_tid=5, entry_tid=7550], [object_number=21, tag_tid=5, entry_tid=7549]]], state=connected] 2024-04-24T00:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=7552], [object_number=23, tag_tid=5, entry_tid=7551], [object_number=22, tag_tid=5, entry_tid=7550], [object_number=21, tag_tid=5, entry_tid=7549]]], state=connected] 2024-04-24T00:41:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=20, tag_tid=5, entry_tid=7552' 2024-04-24T00:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=20, tag_tid=5, entry_tid=7552' 2024-04-24T00:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=7552' '!=' 'object_number=23, tag_tid=5, entry_tid=7391' 2024-04-24T00:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=7552' = 'object_number=23, tag_tid=5, entry_tid=7391' 2024-04-24T00:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=20, tag_tid=5, entry_tid=7552' 2024-04-24T00:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=7552' 2024-04-24T00:41:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=7552' 2024-04-24T00:41:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7552 2024-04-24T00:41:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 7552 2024-04-24T00:41:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=23, tag_tid=5, entry_tid=7671], [object_number=22, tag_tid=5, entry_tid=7670], [object_number=21, tag_tid=5, entry_tid=7669], [object_number=20, tag_tid=5, entry_tid=7668]]], state=connected] 2024-04-24T00:41:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=23, tag_tid=5, entry_tid=7671], [object_number=22, tag_tid=5, entry_tid=7670], [object_number=21, tag_tid=5, entry_tid=7669], [object_number=20, tag_tid=5, entry_tid=7668]]], state=connected] 2024-04-24T00:41:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=23, tag_tid=5, entry_tid=7671' 2024-04-24T00:41:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=23, tag_tid=5, entry_tid=7671' 2024-04-24T00:41:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=23, tag_tid=5, entry_tid=7671' '!=' 'object_number=20, tag_tid=5, entry_tid=7552' 2024-04-24T00:41:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=23, tag_tid=5, entry_tid=7671' = 'object_number=20, tag_tid=5, entry_tid=7552' 2024-04-24T00:41:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=23, tag_tid=5, entry_tid=7671' 2024-04-24T00:41:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=23, tag_tid=5, entry_tid=7671' 2024-04-24T00:41:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=23, tag_tid=5, entry_tid=7671' 2024-04-24T00:41:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7671 2024-04-24T00:41:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 7671 2024-04-24T00:41:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=22, tag_tid=5, entry_tid=7802], [object_number=21, tag_tid=5, entry_tid=7801], [object_number=20, tag_tid=5, entry_tid=7800], [object_number=23, tag_tid=5, entry_tid=7799]]], state=connected] 2024-04-24T00:41:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=22, tag_tid=5, entry_tid=7802], [object_number=21, tag_tid=5, entry_tid=7801], [object_number=20, tag_tid=5, entry_tid=7800], [object_number=23, tag_tid=5, entry_tid=7799]]], state=connected] 2024-04-24T00:41:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=22, tag_tid=5, entry_tid=7802' 2024-04-24T00:41:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=22, tag_tid=5, entry_tid=7802' 2024-04-24T00:41:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=22, tag_tid=5, entry_tid=7802' '!=' 'object_number=23, tag_tid=5, entry_tid=7671' 2024-04-24T00:41:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=22, tag_tid=5, entry_tid=7802' = 'object_number=23, tag_tid=5, entry_tid=7671' 2024-04-24T00:41:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=22, tag_tid=5, entry_tid=7802' 2024-04-24T00:41:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=22, tag_tid=5, entry_tid=7802' 2024-04-24T00:41:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=22, tag_tid=5, entry_tid=7802' 2024-04-24T00:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7802 2024-04-24T00:41:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 7802 2024-04-24T00:41:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=7948], [object_number=23, tag_tid=5, entry_tid=7947], [object_number=22, tag_tid=5, entry_tid=7946], [object_number=21, tag_tid=5, entry_tid=7945]]], state=connected] 2024-04-24T00:41:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=7948], [object_number=23, tag_tid=5, entry_tid=7947], [object_number=22, tag_tid=5, entry_tid=7946], [object_number=21, tag_tid=5, entry_tid=7945]]], state=connected] 2024-04-24T00:41:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=20, tag_tid=5, entry_tid=7948' 2024-04-24T00:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=20, tag_tid=5, entry_tid=7948' 2024-04-24T00:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=7948' '!=' 'object_number=22, tag_tid=5, entry_tid=7802' 2024-04-24T00:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=7948' = 'object_number=22, tag_tid=5, entry_tid=7802' 2024-04-24T00:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=20, tag_tid=5, entry_tid=7948' 2024-04-24T00:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=7948' 2024-04-24T00:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=7948' 2024-04-24T00:41:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7948 2024-04-24T00:41:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 7948 2024-04-24T00:41:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=8096], [object_number=23, tag_tid=5, entry_tid=8095], [object_number=22, tag_tid=5, entry_tid=8094], [object_number=21, tag_tid=5, entry_tid=8093]]], state=connected] 2024-04-24T00:41:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=8096], [object_number=23, tag_tid=5, entry_tid=8095], [object_number=22, tag_tid=5, entry_tid=8094], [object_number=21, tag_tid=5, entry_tid=8093]]], state=connected] 2024-04-24T00:41:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=20, tag_tid=5, entry_tid=8096' 2024-04-24T00:41:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=20, tag_tid=5, entry_tid=8096' 2024-04-24T00:41:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=8096' '!=' 'object_number=20, tag_tid=5, entry_tid=7948' 2024-04-24T00:41:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=8096' = 'object_number=20, tag_tid=5, entry_tid=7948' 2024-04-24T00:41:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=20, tag_tid=5, entry_tid=8096' 2024-04-24T00:41:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=8096' 2024-04-24T00:41:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=8096' 2024-04-24T00:41:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8096 2024-04-24T00:41:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 8096 2024-04-24T00:41:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=22, tag_tid=5, entry_tid=8218], [object_number=21, tag_tid=5, entry_tid=8217], [object_number=20, tag_tid=5, entry_tid=8216], [object_number=23, tag_tid=5, entry_tid=8215]]], state=connected] 2024-04-24T00:41:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=22, tag_tid=5, entry_tid=8218], [object_number=21, tag_tid=5, entry_tid=8217], [object_number=20, tag_tid=5, entry_tid=8216], [object_number=23, tag_tid=5, entry_tid=8215]]], state=connected] 2024-04-24T00:41:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=22, tag_tid=5, entry_tid=8218' 2024-04-24T00:41:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=22, tag_tid=5, entry_tid=8218' 2024-04-24T00:41:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=22, tag_tid=5, entry_tid=8218' '!=' 'object_number=20, tag_tid=5, entry_tid=8096' 2024-04-24T00:41:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=22, tag_tid=5, entry_tid=8218' = 'object_number=20, tag_tid=5, entry_tid=8096' 2024-04-24T00:41:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=22, tag_tid=5, entry_tid=8218' 2024-04-24T00:41:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=22, tag_tid=5, entry_tid=8218' 2024-04-24T00:41:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=22, tag_tid=5, entry_tid=8218' 2024-04-24T00:41:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8218 2024-04-24T00:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 8218 2024-04-24T00:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=8344], [object_number=23, tag_tid=5, entry_tid=8343], [object_number=22, tag_tid=5, entry_tid=8342], [object_number=21, tag_tid=5, entry_tid=8341]]], state=connected] 2024-04-24T00:41:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=20, tag_tid=5, entry_tid=8344], [object_number=23, tag_tid=5, entry_tid=8343], [object_number=22, tag_tid=5, entry_tid=8342], [object_number=21, tag_tid=5, entry_tid=8341]]], state=connected] 2024-04-24T00:41:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=20, tag_tid=5, entry_tid=8344' 2024-04-24T00:41:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=20, tag_tid=5, entry_tid=8344' 2024-04-24T00:41:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=8344' '!=' 'object_number=22, tag_tid=5, entry_tid=8218' 2024-04-24T00:41:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=20, tag_tid=5, entry_tid=8344' = 'object_number=22, tag_tid=5, entry_tid=8218' 2024-04-24T00:41:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=20, tag_tid=5, entry_tid=8344' 2024-04-24T00:41:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=8344' 2024-04-24T00:41:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=20, tag_tid=5, entry_tid=8344' 2024-04-24T00:41:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8344 2024-04-24T00:41:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 8344 2024-04-24T00:41:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=23, tag_tid=5, entry_tid=8463], [object_number=22, tag_tid=5, entry_tid=8462], [object_number=21, tag_tid=5, entry_tid=8461], [object_number=20, tag_tid=5, entry_tid=8460]]], state=connected] 2024-04-24T00:41:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=23, tag_tid=5, entry_tid=8463], [object_number=22, tag_tid=5, entry_tid=8462], [object_number=21, tag_tid=5, entry_tid=8461], [object_number=20, tag_tid=5, entry_tid=8460]]], state=connected] 2024-04-24T00:41:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=23, tag_tid=5, entry_tid=8463' 2024-04-24T00:41:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=23, tag_tid=5, entry_tid=8463' 2024-04-24T00:41:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=23, tag_tid=5, entry_tid=8463' '!=' 'object_number=20, tag_tid=5, entry_tid=8344' 2024-04-24T00:41:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=23, tag_tid=5, entry_tid=8463' = 'object_number=20, tag_tid=5, entry_tid=8344' 2024-04-24T00:41:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=23, tag_tid=5, entry_tid=8463' 2024-04-24T00:41:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=23, tag_tid=5, entry_tid=8463' 2024-04-24T00:41:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=23, tag_tid=5, entry_tid=8463' 2024-04-24T00:41:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8463 2024-04-24T00:41:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 8463 2024-04-24T00:41:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570], [object_number=21, tag_tid=5, entry_tid=8569], [object_number=20, tag_tid=5, entry_tid=8568]]], state=connected] 2024-04-24T00:41:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9898377553988543) > weight (0.25) 2024-04-24T00:41:58.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6362185603773725) > weight (0.25) 2024-04-24T00:41:58.676 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.49799902002294893) > weight (0.25) 2024-04-24T00:41:58.676 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7695832020300198) > weight (0.25) 2024-04-24T00:41:58.676 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-04-24T00:41:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570], [object_number=21, tag_tid=5, entry_tid=8569], [object_number=20, tag_tid=5, entry_tid=8568]]], state=connected] 2024-04-24T00:41:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=23, tag_tid=5, entry_tid=8571' 2024-04-24T00:41:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=23, tag_tid=5, entry_tid=8571' 2024-04-24T00:41:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=23, tag_tid=5, entry_tid=8571' '!=' 'object_number=23, tag_tid=5, entry_tid=8463' 2024-04-24T00:41:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:41:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=23, tag_tid=5, entry_tid=8571' = 'object_number=23, tag_tid=5, entry_tid=8463' 2024-04-24T00:41:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=23, tag_tid=5, entry_tid=8571' 2024-04-24T00:41:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=6 2024-04-24T00:41:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=23, tag_tid=5, entry_tid=8571' 2024-04-24T00:41:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:41:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=5 2024-04-24T00:41:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:41:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=23, tag_tid=5, entry_tid=8571' 2024-04-24T00:41:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:41:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:41:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8571 2024-04-24T00:41:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 5 -a 1 -le 8571 2024-04-24T00:41:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:41:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:41:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:41:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:41:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:41:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:41:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:41:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:41:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:41:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:41:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:41:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:41:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:41:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:41:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:41:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:41:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:41:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:41:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:41:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:41:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:41:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:41:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:41:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:41:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:41:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:41:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:41:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:41:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:41:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:41:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:41:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:41:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:41:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:41:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:41:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:41:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:41:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:41:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:41:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:41:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:41:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:41:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:41:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:41:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:41:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:41:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:41:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:41:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 5 2024-04-24T00:41:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 5 2024-04-24T00:41:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:41:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=21, tag_tid=6, entry_tid=1], [object_number=20, tag_tid=6, entry_tid=0], [object_number=23, tag_tid=5, entry_tid=8571], [object_number=22, tag_tid=5, entry_tid=8570]]], state=connected] 2024-04-24T00:41:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=21, tag_tid=6, entry_tid=1' -a 'object_number=21, tag_tid=6, entry_tid=1' = 'object_number=21, tag_tid=6, entry_tid=1' 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:41:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:41:59.678 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.37761696435568604) > weight (0.25) 2024-04-24T00:41:59.678 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8104193369108916) > weight (0.25) 2024-04-24T00:41:59.678 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.634584394187827) > weight (0.25) 2024-04-24T00:41:59.678 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4856537601402007) > weight (0.25) 2024-04-24T00:41:59.678 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-04-24T00:42:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-04-24T00:42:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:42:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:42:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 3 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:40:44 2024 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:41:30 2024 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:42:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:42:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:42:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:42:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap1-clean ']' 2024-04-24T00:42:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-24T00:42:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1-clean 2024-04-24T00:42:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:42:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:42:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:42:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:42:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:42:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap2 2024-04-24T00:42:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2 2024-04-24T00:42:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:42:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:42:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:42:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:42:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:02.973 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:42:02.971+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:42:02.974 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:42:02.971+0000 7efd524cf640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:42:02.975 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:42:02.971+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:42:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:42:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:42:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap2-clean 2024-04-24T00:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-04-24T00:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2-clean 2024-04-24T00:42:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:42:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:42:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:42:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:42:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:42:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T00:42:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:42:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:42:04.401 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:42:04.396+0000 7ff573346640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 5 2024-04-24T00:42:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 27616256 bytes 2024-04-24T00:42:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 48934912 bytes 2024-04-24T00:42:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 64260096 bytes 2024-04-24T00:42:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 76854272 bytes 2024-04-24T00:42:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 133381120 bytes 2024-04-24T00:42:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 153729024 bytes 2024-04-24T00:42:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 205450240 bytes 2024-04-24T00:42:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 242402304 bytes 2024-04-24T00:42:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 276476928 bytes 2024-04-24T00:42:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 319525888 bytes 2024-04-24T00:42:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 393440256 bytes 2024-04-24T00:42:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 398113792 bytes 2024-04-24T00:42:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:42:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:42:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:42:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:42:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:42:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:42:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:42:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:42:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:42:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:42:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:42:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:42:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":3365172.27,"entries_behind_primary":476,"entries_per_second":209.43,"non_primary_position":{"entry_tid":8096,"object_number":20,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":21,"tag_tid":6},"seconds_until_synced":2} 2024-04-24T00:42:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T00:42:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:41:56 2024-04-24T00:42:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:42:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:42:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:42:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:42:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:41:56' 2024-04-24T00:42:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:42:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:42:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:42:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":3365172.27,"entries_behind_primary":476,"entries_per_second":209.43,"non_primary_position":{"entry_tid":8096,"object_number":20,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":21,"tag_tid":6},"seconds_until_synced":2} 2024-04-24T00:42:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:42:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:42:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:42:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:42:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:42:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T00:42:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:42:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:42:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:41:56' 2024-04-24T00:42:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:41:56' 2024-04-24T00:42:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:42:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:41:56' 2024-04-24T00:42:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:42:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:42:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:42:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142114271mirror.1smithi13419.0.0-2424-gb98aab6ctrueOK42194274mirror.3smithi13419.0.0-2424-gb98aab6cfalseOK42594312mirror.2smithi13419.0.0-2424-gb98aab6cfalseOK43304333mirror.0smithi13419.0.0-2424-gb98aab6cfalseOKtestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":3365172.27,"entries_behind_primary":476,"entries_per_second":209.43,"non_primary_position":{"entry_tid":8096,"object_number":20,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":21,"tag_tid":6},"seconds_until_synced":2}42114271mirror.1smithi1342024-04-24 00:41:56cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:41:56' 2024-04-24T00:42:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:42:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:42:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:41:56' 2024-04-24T00:42:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:42:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:42:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:42:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:42:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:42:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3 2024-04-24T00:42:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap3 2024-04-24T00:42:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:42:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3 2024-04-24T00:42:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:42:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:42:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:42:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:42:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:42:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:42:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:42:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:42:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:42:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:42:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:42:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:42:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:42:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:42:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:42:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:42:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:42:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:42:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:42:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:42:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:42:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:42:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:42:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:42:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:42:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:42:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:42:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:42:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:42:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:42:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:42:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:42:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:42:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:42:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:42:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:42:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:42:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:42:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=8488], [object_number=31, tag_tid=8, entry_tid=8487], [object_number=30, tag_tid=8, entry_tid=8486], [object_number=29, tag_tid=8, entry_tid=8485]]], state=connected] 2024-04-24T00:42:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=8488], [object_number=31, tag_tid=8, entry_tid=8487], [object_number=30, tag_tid=8, entry_tid=8486], [object_number=29, tag_tid=8, entry_tid=8485]]], state=connected] 2024-04-24T00:42:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=28, tag_tid=8, entry_tid=8488' 2024-04-24T00:42:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=28, tag_tid=8, entry_tid=8488' 2024-04-24T00:42:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=8488' '!=' '' 2024-04-24T00:42:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=8488' = '' 2024-04-24T00:42:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=28, tag_tid=8, entry_tid=8488' 2024-04-24T00:42:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=8488' 2024-04-24T00:42:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=8488' 2024-04-24T00:42:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8488 2024-04-24T00:42:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 8488 2024-04-24T00:42:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=30, tag_tid=8, entry_tid=8602], [object_number=29, tag_tid=8, entry_tid=8601], [object_number=28, tag_tid=8, entry_tid=8600], [object_number=31, tag_tid=8, entry_tid=8599]]], state=connected] 2024-04-24T00:42:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=30, tag_tid=8, entry_tid=8602], [object_number=29, tag_tid=8, entry_tid=8601], [object_number=28, tag_tid=8, entry_tid=8600], [object_number=31, tag_tid=8, entry_tid=8599]]], state=connected] 2024-04-24T00:42:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=30, tag_tid=8, entry_tid=8602' 2024-04-24T00:42:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=30, tag_tid=8, entry_tid=8602' 2024-04-24T00:42:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=30, tag_tid=8, entry_tid=8602' '!=' 'object_number=28, tag_tid=8, entry_tid=8488' 2024-04-24T00:42:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=30, tag_tid=8, entry_tid=8602' = 'object_number=28, tag_tid=8, entry_tid=8488' 2024-04-24T00:42:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=30, tag_tid=8, entry_tid=8602' 2024-04-24T00:42:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=30, tag_tid=8, entry_tid=8602' 2024-04-24T00:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=30, tag_tid=8, entry_tid=8602' 2024-04-24T00:42:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8602 2024-04-24T00:42:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 8602 2024-04-24T00:42:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:53.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=30, tag_tid=8, entry_tid=8710], [object_number=29, tag_tid=8, entry_tid=8709], [object_number=28, tag_tid=8, entry_tid=8708], [object_number=31, tag_tid=8, entry_tid=8707]]], state=connected] 2024-04-24T00:42:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=30, tag_tid=8, entry_tid=8710], [object_number=29, tag_tid=8, entry_tid=8709], [object_number=28, tag_tid=8, entry_tid=8708], [object_number=31, tag_tid=8, entry_tid=8707]]], state=connected] 2024-04-24T00:42:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=30, tag_tid=8, entry_tid=8710' 2024-04-24T00:42:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=30, tag_tid=8, entry_tid=8710' 2024-04-24T00:42:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=30, tag_tid=8, entry_tid=8710' '!=' 'object_number=30, tag_tid=8, entry_tid=8602' 2024-04-24T00:42:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=30, tag_tid=8, entry_tid=8710' = 'object_number=30, tag_tid=8, entry_tid=8602' 2024-04-24T00:42:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=30, tag_tid=8, entry_tid=8710' 2024-04-24T00:42:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=30, tag_tid=8, entry_tid=8710' 2024-04-24T00:42:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=30, tag_tid=8, entry_tid=8710' 2024-04-24T00:42:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8710 2024-04-24T00:42:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 8710 2024-04-24T00:42:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=29, tag_tid=8, entry_tid=8841], [object_number=28, tag_tid=8, entry_tid=8840], [object_number=31, tag_tid=8, entry_tid=8839], [object_number=30, tag_tid=8, entry_tid=8838]]], state=connected] 2024-04-24T00:42:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=29, tag_tid=8, entry_tid=8841], [object_number=28, tag_tid=8, entry_tid=8840], [object_number=31, tag_tid=8, entry_tid=8839], [object_number=30, tag_tid=8, entry_tid=8838]]], state=connected] 2024-04-24T00:42:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=29, tag_tid=8, entry_tid=8841' 2024-04-24T00:42:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=29, tag_tid=8, entry_tid=8841' 2024-04-24T00:42:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=29, tag_tid=8, entry_tid=8841' '!=' 'object_number=30, tag_tid=8, entry_tid=8710' 2024-04-24T00:42:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=29, tag_tid=8, entry_tid=8841' = 'object_number=30, tag_tid=8, entry_tid=8710' 2024-04-24T00:42:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=29, tag_tid=8, entry_tid=8841' 2024-04-24T00:42:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=29, tag_tid=8, entry_tid=8841' 2024-04-24T00:42:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=29, tag_tid=8, entry_tid=8841' 2024-04-24T00:42:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8841 2024-04-24T00:42:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 8841 2024-04-24T00:42:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=8972], [object_number=31, tag_tid=8, entry_tid=8971], [object_number=30, tag_tid=8, entry_tid=8970], [object_number=29, tag_tid=8, entry_tid=8969]]], state=connected] 2024-04-24T00:42:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=8972], [object_number=31, tag_tid=8, entry_tid=8971], [object_number=30, tag_tid=8, entry_tid=8970], [object_number=29, tag_tid=8, entry_tid=8969]]], state=connected] 2024-04-24T00:42:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=28, tag_tid=8, entry_tid=8972' 2024-04-24T00:42:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=28, tag_tid=8, entry_tid=8972' 2024-04-24T00:42:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=8972' '!=' 'object_number=29, tag_tid=8, entry_tid=8841' 2024-04-24T00:42:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=8972' = 'object_number=29, tag_tid=8, entry_tid=8841' 2024-04-24T00:42:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=28, tag_tid=8, entry_tid=8972' 2024-04-24T00:42:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=8972' 2024-04-24T00:42:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=8972' 2024-04-24T00:42:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8972 2024-04-24T00:42:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 8972 2024-04-24T00:42:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=9096], [object_number=31, tag_tid=8, entry_tid=9095], [object_number=30, tag_tid=8, entry_tid=9094], [object_number=29, tag_tid=8, entry_tid=9093]]], state=connected] 2024-04-24T00:42:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=9096], [object_number=31, tag_tid=8, entry_tid=9095], [object_number=30, tag_tid=8, entry_tid=9094], [object_number=29, tag_tid=8, entry_tid=9093]]], state=connected] 2024-04-24T00:42:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=28, tag_tid=8, entry_tid=9096' 2024-04-24T00:42:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=28, tag_tid=8, entry_tid=9096' 2024-04-24T00:42:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=9096' '!=' 'object_number=28, tag_tid=8, entry_tid=8972' 2024-04-24T00:42:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=9096' = 'object_number=28, tag_tid=8, entry_tid=8972' 2024-04-24T00:42:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=28, tag_tid=8, entry_tid=9096' 2024-04-24T00:42:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=9096' 2024-04-24T00:42:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=9096' 2024-04-24T00:42:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9096 2024-04-24T00:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 9096 2024-04-24T00:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=9220], [object_number=31, tag_tid=8, entry_tid=9219], [object_number=30, tag_tid=8, entry_tid=9218], [object_number=29, tag_tid=8, entry_tid=9217]]], state=connected] 2024-04-24T00:42:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=9220], [object_number=31, tag_tid=8, entry_tid=9219], [object_number=30, tag_tid=8, entry_tid=9218], [object_number=29, tag_tid=8, entry_tid=9217]]], state=connected] 2024-04-24T00:42:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=28, tag_tid=8, entry_tid=9220' 2024-04-24T00:42:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=28, tag_tid=8, entry_tid=9220' 2024-04-24T00:42:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=9220' '!=' 'object_number=28, tag_tid=8, entry_tid=9096' 2024-04-24T00:42:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=9220' = 'object_number=28, tag_tid=8, entry_tid=9096' 2024-04-24T00:42:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=28, tag_tid=8, entry_tid=9220' 2024-04-24T00:42:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=9220' 2024-04-24T00:42:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=9220' 2024-04-24T00:42:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9220 2024-04-24T00:42:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 9220 2024-04-24T00:42:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=30, tag_tid=8, entry_tid=9350], [object_number=29, tag_tid=8, entry_tid=9349], [object_number=28, tag_tid=8, entry_tid=9348], [object_number=31, tag_tid=8, entry_tid=9347]]], state=connected] 2024-04-24T00:42:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=30, tag_tid=8, entry_tid=9350], [object_number=29, tag_tid=8, entry_tid=9349], [object_number=28, tag_tid=8, entry_tid=9348], [object_number=31, tag_tid=8, entry_tid=9347]]], state=connected] 2024-04-24T00:42:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=30, tag_tid=8, entry_tid=9350' 2024-04-24T00:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=30, tag_tid=8, entry_tid=9350' 2024-04-24T00:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=30, tag_tid=8, entry_tid=9350' '!=' 'object_number=28, tag_tid=8, entry_tid=9220' 2024-04-24T00:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=30, tag_tid=8, entry_tid=9350' = 'object_number=28, tag_tid=8, entry_tid=9220' 2024-04-24T00:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=30, tag_tid=8, entry_tid=9350' 2024-04-24T00:42:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=30, tag_tid=8, entry_tid=9350' 2024-04-24T00:42:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=30, tag_tid=8, entry_tid=9350' 2024-04-24T00:42:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9350 2024-04-24T00:42:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 9350 2024-04-24T00:42:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=30, tag_tid=8, entry_tid=9478], [object_number=29, tag_tid=8, entry_tid=9477], [object_number=28, tag_tid=8, entry_tid=9476], [object_number=31, tag_tid=8, entry_tid=9475]]], state=connected] 2024-04-24T00:42:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=30, tag_tid=8, entry_tid=9478], [object_number=29, tag_tid=8, entry_tid=9477], [object_number=28, tag_tid=8, entry_tid=9476], [object_number=31, tag_tid=8, entry_tid=9475]]], state=connected] 2024-04-24T00:42:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=30, tag_tid=8, entry_tid=9478' 2024-04-24T00:42:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=30, tag_tid=8, entry_tid=9478' 2024-04-24T00:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=30, tag_tid=8, entry_tid=9478' '!=' 'object_number=30, tag_tid=8, entry_tid=9350' 2024-04-24T00:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=30, tag_tid=8, entry_tid=9478' = 'object_number=30, tag_tid=8, entry_tid=9350' 2024-04-24T00:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=30, tag_tid=8, entry_tid=9478' 2024-04-24T00:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=30, tag_tid=8, entry_tid=9478' 2024-04-24T00:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=30, tag_tid=8, entry_tid=9478' 2024-04-24T00:42:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9478 2024-04-24T00:42:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 9478 2024-04-24T00:42:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=9600], [object_number=31, tag_tid=8, entry_tid=9599], [object_number=30, tag_tid=8, entry_tid=9598], [object_number=29, tag_tid=8, entry_tid=9597]]], state=connected] 2024-04-24T00:42:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=9600], [object_number=31, tag_tid=8, entry_tid=9599], [object_number=30, tag_tid=8, entry_tid=9598], [object_number=29, tag_tid=8, entry_tid=9597]]], state=connected] 2024-04-24T00:42:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=28, tag_tid=8, entry_tid=9600' 2024-04-24T00:42:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=28, tag_tid=8, entry_tid=9600' 2024-04-24T00:42:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=9600' '!=' 'object_number=30, tag_tid=8, entry_tid=9478' 2024-04-24T00:42:58.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:58.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=9600' = 'object_number=30, tag_tid=8, entry_tid=9478' 2024-04-24T00:42:58.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=28, tag_tid=8, entry_tid=9600' 2024-04-24T00:42:58.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=9600' 2024-04-24T00:42:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:58.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=9600' 2024-04-24T00:42:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9600 2024-04-24T00:42:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 9600 2024-04-24T00:42:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=29, tag_tid=8, entry_tid=9785], [object_number=28, tag_tid=8, entry_tid=9784], [object_number=31, tag_tid=8, entry_tid=9783], [object_number=30, tag_tid=8, entry_tid=9782]]], state=connected] 2024-04-24T00:42:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=29, tag_tid=8, entry_tid=9785], [object_number=28, tag_tid=8, entry_tid=9784], [object_number=31, tag_tid=8, entry_tid=9783], [object_number=30, tag_tid=8, entry_tid=9782]]], state=connected] 2024-04-24T00:42:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=29, tag_tid=8, entry_tid=9785' 2024-04-24T00:42:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=29, tag_tid=8, entry_tid=9785' 2024-04-24T00:42:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=29, tag_tid=8, entry_tid=9785' '!=' 'object_number=28, tag_tid=8, entry_tid=9600' 2024-04-24T00:42:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=29, tag_tid=8, entry_tid=9785' = 'object_number=28, tag_tid=8, entry_tid=9600' 2024-04-24T00:42:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=29, tag_tid=8, entry_tid=9785' 2024-04-24T00:42:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=29, tag_tid=8, entry_tid=9785' 2024-04-24T00:42:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=29, tag_tid=8, entry_tid=9785' 2024-04-24T00:42:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9785 2024-04-24T00:42:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 9785 2024-04-24T00:42:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=9916], [object_number=31, tag_tid=8, entry_tid=9915], [object_number=30, tag_tid=8, entry_tid=9914], [object_number=29, tag_tid=8, entry_tid=9913]]], state=connected] 2024-04-24T00:42:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:42:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:42:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:42:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:42:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:42:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:42:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:42:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=28, tag_tid=8, entry_tid=9916], [object_number=31, tag_tid=8, entry_tid=9915], [object_number=30, tag_tid=8, entry_tid=9914], [object_number=29, tag_tid=8, entry_tid=9913]]], state=connected] 2024-04-24T00:42:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=28, tag_tid=8, entry_tid=9916' 2024-04-24T00:42:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=28, tag_tid=8, entry_tid=9916' 2024-04-24T00:42:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=9916' '!=' 'object_number=29, tag_tid=8, entry_tid=9785' 2024-04-24T00:42:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:42:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=28, tag_tid=8, entry_tid=9916' = 'object_number=29, tag_tid=8, entry_tid=9785' 2024-04-24T00:42:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=28, tag_tid=8, entry_tid=9916' 2024-04-24T00:42:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:42:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=9916' 2024-04-24T00:42:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:42:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:42:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:42:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:42:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=28, tag_tid=8, entry_tid=9916' 2024-04-24T00:42:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:42:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:42:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9916 2024-04-24T00:42:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 9916 2024-04-24T00:42:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:42:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:42:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:42:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:42:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:42:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:42:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:42:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:42:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:42:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:42:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:42:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:42:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:42:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:42:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:42:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:42:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:42:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:42:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:42:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:42:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:42:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:42:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:42:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:42:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:42:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:42:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:42:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:42:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:42:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:42:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:42:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:42:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:42:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:42:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:42:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:42:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:42:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:42:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:42:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:42:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:42:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:42:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:42:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:42:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:42:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:42:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:42:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:42:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:42:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:42:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:42:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:42:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:42:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:42:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:42:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:42:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:42:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:42:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:42:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:43:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10018], [object_number=29, tag_tid=8, entry_tid=10017], [object_number=28, tag_tid=8, entry_tid=10016], [object_number=31, tag_tid=8, entry_tid=10015]]], state=connected] 2024-04-24T00:43:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 7 2024-04-24T00:43:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10018], [object_number=29, tag_tid=8, entry_tid=10017], [object_number=28, tag_tid=8, entry_tid=10016], [object_number=31, tag_tid=8, entry_tid=10015]]], state=connected] 2024-04-24T00:43:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=34, tag_tid=8, entry_tid=10018' 2024-04-24T00:43:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=34, tag_tid=8, entry_tid=10018' 2024-04-24T00:43:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10018' '!=' 'object_number=28, tag_tid=8, entry_tid=9916' 2024-04-24T00:43:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10018' = 'object_number=28, tag_tid=8, entry_tid=9916' 2024-04-24T00:43:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=34, tag_tid=8, entry_tid=10018' 2024-04-24T00:43:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10018' 2024-04-24T00:43:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10018' 2024-04-24T00:43:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10018 2024-04-24T00:43:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 10018 2024-04-24T00:43:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10122], [object_number=33, tag_tid=8, entry_tid=10121], [object_number=32, tag_tid=8, entry_tid=10120], [object_number=35, tag_tid=8, entry_tid=10119]]], state=connected] 2024-04-24T00:43:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10122], [object_number=33, tag_tid=8, entry_tid=10121], [object_number=32, tag_tid=8, entry_tid=10120], [object_number=35, tag_tid=8, entry_tid=10119]]], state=connected] 2024-04-24T00:43:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=34, tag_tid=8, entry_tid=10122' 2024-04-24T00:43:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=34, tag_tid=8, entry_tid=10122' 2024-04-24T00:43:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10122' '!=' 'object_number=34, tag_tid=8, entry_tid=10018' 2024-04-24T00:43:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10122' = 'object_number=34, tag_tid=8, entry_tid=10018' 2024-04-24T00:43:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=34, tag_tid=8, entry_tid=10122' 2024-04-24T00:43:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10122' 2024-04-24T00:43:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10122' 2024-04-24T00:43:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10122 2024-04-24T00:43:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 10122 2024-04-24T00:43:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:01.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=10255], [object_number=34, tag_tid=8, entry_tid=10254], [object_number=33, tag_tid=8, entry_tid=10253], [object_number=32, tag_tid=8, entry_tid=10252]]], state=connected] 2024-04-24T00:43:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=10255], [object_number=34, tag_tid=8, entry_tid=10254], [object_number=33, tag_tid=8, entry_tid=10253], [object_number=32, tag_tid=8, entry_tid=10252]]], state=connected] 2024-04-24T00:43:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=35, tag_tid=8, entry_tid=10255' 2024-04-24T00:43:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=35, tag_tid=8, entry_tid=10255' 2024-04-24T00:43:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=10255' '!=' 'object_number=34, tag_tid=8, entry_tid=10122' 2024-04-24T00:43:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=10255' = 'object_number=34, tag_tid=8, entry_tid=10122' 2024-04-24T00:43:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=35, tag_tid=8, entry_tid=10255' 2024-04-24T00:43:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=10255' 2024-04-24T00:43:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=10255' 2024-04-24T00:43:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10255 2024-04-24T00:43:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 10255 2024-04-24T00:43:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10386], [object_number=33, tag_tid=8, entry_tid=10385], [object_number=32, tag_tid=8, entry_tid=10384], [object_number=35, tag_tid=8, entry_tid=10383]]], state=connected] 2024-04-24T00:43:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10386], [object_number=33, tag_tid=8, entry_tid=10385], [object_number=32, tag_tid=8, entry_tid=10384], [object_number=35, tag_tid=8, entry_tid=10383]]], state=connected] 2024-04-24T00:43:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=34, tag_tid=8, entry_tid=10386' 2024-04-24T00:43:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=34, tag_tid=8, entry_tid=10386' 2024-04-24T00:43:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10386' '!=' 'object_number=35, tag_tid=8, entry_tid=10255' 2024-04-24T00:43:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10386' = 'object_number=35, tag_tid=8, entry_tid=10255' 2024-04-24T00:43:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=34, tag_tid=8, entry_tid=10386' 2024-04-24T00:43:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10386' 2024-04-24T00:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10386' 2024-04-24T00:43:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10386 2024-04-24T00:43:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 10386 2024-04-24T00:43:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10518], [object_number=33, tag_tid=8, entry_tid=10517], [object_number=32, tag_tid=8, entry_tid=10516], [object_number=35, tag_tid=8, entry_tid=10515]]], state=connected] 2024-04-24T00:43:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10518], [object_number=33, tag_tid=8, entry_tid=10517], [object_number=32, tag_tid=8, entry_tid=10516], [object_number=35, tag_tid=8, entry_tid=10515]]], state=connected] 2024-04-24T00:43:02.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=34, tag_tid=8, entry_tid=10518' 2024-04-24T00:43:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=34, tag_tid=8, entry_tid=10518' 2024-04-24T00:43:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10518' '!=' 'object_number=34, tag_tid=8, entry_tid=10386' 2024-04-24T00:43:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10518' = 'object_number=34, tag_tid=8, entry_tid=10386' 2024-04-24T00:43:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=34, tag_tid=8, entry_tid=10518' 2024-04-24T00:43:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10518' 2024-04-24T00:43:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10518' 2024-04-24T00:43:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10518 2024-04-24T00:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 10518 2024-04-24T00:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10642], [object_number=33, tag_tid=8, entry_tid=10641], [object_number=32, tag_tid=8, entry_tid=10640], [object_number=35, tag_tid=8, entry_tid=10639]]], state=connected] 2024-04-24T00:43:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10642], [object_number=33, tag_tid=8, entry_tid=10641], [object_number=32, tag_tid=8, entry_tid=10640], [object_number=35, tag_tid=8, entry_tid=10639]]], state=connected] 2024-04-24T00:43:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=34, tag_tid=8, entry_tid=10642' 2024-04-24T00:43:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=34, tag_tid=8, entry_tid=10642' 2024-04-24T00:43:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10642' '!=' 'object_number=34, tag_tid=8, entry_tid=10518' 2024-04-24T00:43:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10642' = 'object_number=34, tag_tid=8, entry_tid=10518' 2024-04-24T00:43:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=34, tag_tid=8, entry_tid=10642' 2024-04-24T00:43:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10642' 2024-04-24T00:43:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10642' 2024-04-24T00:43:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10642 2024-04-24T00:43:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 10642 2024-04-24T00:43:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=10773], [object_number=32, tag_tid=8, entry_tid=10772], [object_number=35, tag_tid=8, entry_tid=10771], [object_number=34, tag_tid=8, entry_tid=10770]]], state=connected] 2024-04-24T00:43:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=10773], [object_number=32, tag_tid=8, entry_tid=10772], [object_number=35, tag_tid=8, entry_tid=10771], [object_number=34, tag_tid=8, entry_tid=10770]]], state=connected] 2024-04-24T00:43:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=10773' 2024-04-24T00:43:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=10773' 2024-04-24T00:43:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=10773' '!=' 'object_number=34, tag_tid=8, entry_tid=10642' 2024-04-24T00:43:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=10773' = 'object_number=34, tag_tid=8, entry_tid=10642' 2024-04-24T00:43:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=10773' 2024-04-24T00:43:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=10773' 2024-04-24T00:43:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=10773' 2024-04-24T00:43:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10773 2024-04-24T00:43:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 10773 2024-04-24T00:43:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10898], [object_number=33, tag_tid=8, entry_tid=10897], [object_number=32, tag_tid=8, entry_tid=10896], [object_number=35, tag_tid=8, entry_tid=10895]]], state=connected] 2024-04-24T00:43:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=10898], [object_number=33, tag_tid=8, entry_tid=10897], [object_number=32, tag_tid=8, entry_tid=10896], [object_number=35, tag_tid=8, entry_tid=10895]]], state=connected] 2024-04-24T00:43:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=34, tag_tid=8, entry_tid=10898' 2024-04-24T00:43:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=34, tag_tid=8, entry_tid=10898' 2024-04-24T00:43:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10898' '!=' 'object_number=33, tag_tid=8, entry_tid=10773' 2024-04-24T00:43:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=10898' = 'object_number=33, tag_tid=8, entry_tid=10773' 2024-04-24T00:43:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=34, tag_tid=8, entry_tid=10898' 2024-04-24T00:43:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10898' 2024-04-24T00:43:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=10898' 2024-04-24T00:43:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10898 2024-04-24T00:43:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 10898 2024-04-24T00:43:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=11027], [object_number=34, tag_tid=8, entry_tid=11026], [object_number=33, tag_tid=8, entry_tid=11025], [object_number=32, tag_tid=8, entry_tid=11024]]], state=connected] 2024-04-24T00:43:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=11027], [object_number=34, tag_tid=8, entry_tid=11026], [object_number=33, tag_tid=8, entry_tid=11025], [object_number=32, tag_tid=8, entry_tid=11024]]], state=connected] 2024-04-24T00:43:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=35, tag_tid=8, entry_tid=11027' 2024-04-24T00:43:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=35, tag_tid=8, entry_tid=11027' 2024-04-24T00:43:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=11027' '!=' 'object_number=34, tag_tid=8, entry_tid=10898' 2024-04-24T00:43:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=11027' = 'object_number=34, tag_tid=8, entry_tid=10898' 2024-04-24T00:43:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=35, tag_tid=8, entry_tid=11027' 2024-04-24T00:43:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=11027' 2024-04-24T00:43:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=11027' 2024-04-24T00:43:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11027 2024-04-24T00:43:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 11027 2024-04-24T00:43:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=11157], [object_number=32, tag_tid=8, entry_tid=11156], [object_number=35, tag_tid=8, entry_tid=11155], [object_number=34, tag_tid=8, entry_tid=11154]]], state=connected] 2024-04-24T00:43:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=11157], [object_number=32, tag_tid=8, entry_tid=11156], [object_number=35, tag_tid=8, entry_tid=11155], [object_number=34, tag_tid=8, entry_tid=11154]]], state=connected] 2024-04-24T00:43:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=11157' 2024-04-24T00:43:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=11157' 2024-04-24T00:43:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=11157' '!=' 'object_number=35, tag_tid=8, entry_tid=11027' 2024-04-24T00:43:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=11157' = 'object_number=35, tag_tid=8, entry_tid=11027' 2024-04-24T00:43:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=11157' 2024-04-24T00:43:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=11157' 2024-04-24T00:43:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=11157' 2024-04-24T00:43:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11157 2024-04-24T00:43:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 11157 2024-04-24T00:43:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=11290], [object_number=33, tag_tid=8, entry_tid=11289], [object_number=32, tag_tid=8, entry_tid=11288], [object_number=35, tag_tid=8, entry_tid=11287]]], state=connected] 2024-04-24T00:43:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=11290], [object_number=33, tag_tid=8, entry_tid=11289], [object_number=32, tag_tid=8, entry_tid=11288], [object_number=35, tag_tid=8, entry_tid=11287]]], state=connected] 2024-04-24T00:43:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=34, tag_tid=8, entry_tid=11290' 2024-04-24T00:43:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=34, tag_tid=8, entry_tid=11290' 2024-04-24T00:43:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=11290' '!=' 'object_number=33, tag_tid=8, entry_tid=11157' 2024-04-24T00:43:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=11290' = 'object_number=33, tag_tid=8, entry_tid=11157' 2024-04-24T00:43:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=34, tag_tid=8, entry_tid=11290' 2024-04-24T00:43:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=11290' 2024-04-24T00:43:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=11290' 2024-04-24T00:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11290 2024-04-24T00:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 11290 2024-04-24T00:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=11417], [object_number=32, tag_tid=8, entry_tid=11416], [object_number=35, tag_tid=8, entry_tid=11415], [object_number=34, tag_tid=8, entry_tid=11414]]], state=connected] 2024-04-24T00:43:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=11417], [object_number=32, tag_tid=8, entry_tid=11416], [object_number=35, tag_tid=8, entry_tid=11415], [object_number=34, tag_tid=8, entry_tid=11414]]], state=connected] 2024-04-24T00:43:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=11417' 2024-04-24T00:43:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=11417' 2024-04-24T00:43:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=11417' '!=' 'object_number=34, tag_tid=8, entry_tid=11290' 2024-04-24T00:43:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=11417' = 'object_number=34, tag_tid=8, entry_tid=11290' 2024-04-24T00:43:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=11417' 2024-04-24T00:43:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=11417' 2024-04-24T00:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=11417' 2024-04-24T00:43:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11417 2024-04-24T00:43:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 11417 2024-04-24T00:43:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=11541], [object_number=32, tag_tid=8, entry_tid=11540], [object_number=35, tag_tid=8, entry_tid=11539], [object_number=34, tag_tid=8, entry_tid=11538]]], state=connected] 2024-04-24T00:43:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=11541], [object_number=32, tag_tid=8, entry_tid=11540], [object_number=35, tag_tid=8, entry_tid=11539], [object_number=34, tag_tid=8, entry_tid=11538]]], state=connected] 2024-04-24T00:43:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=11541' 2024-04-24T00:43:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=11541' 2024-04-24T00:43:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=11541' '!=' 'object_number=33, tag_tid=8, entry_tid=11417' 2024-04-24T00:43:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=11541' = 'object_number=33, tag_tid=8, entry_tid=11417' 2024-04-24T00:43:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=11541' 2024-04-24T00:43:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=11541' 2024-04-24T00:43:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=11541' 2024-04-24T00:43:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11541 2024-04-24T00:43:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 11541 2024-04-24T00:43:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=11658], [object_number=33, tag_tid=8, entry_tid=11657], [object_number=32, tag_tid=8, entry_tid=11656], [object_number=35, tag_tid=8, entry_tid=11655]]], state=connected] 2024-04-24T00:43:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=11658], [object_number=33, tag_tid=8, entry_tid=11657], [object_number=32, tag_tid=8, entry_tid=11656], [object_number=35, tag_tid=8, entry_tid=11655]]], state=connected] 2024-04-24T00:43:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=34, tag_tid=8, entry_tid=11658' 2024-04-24T00:43:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=34, tag_tid=8, entry_tid=11658' 2024-04-24T00:43:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=11658' '!=' 'object_number=33, tag_tid=8, entry_tid=11541' 2024-04-24T00:43:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=11658' = 'object_number=33, tag_tid=8, entry_tid=11541' 2024-04-24T00:43:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=34, tag_tid=8, entry_tid=11658' 2024-04-24T00:43:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=11658' 2024-04-24T00:43:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=11658' 2024-04-24T00:43:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11658 2024-04-24T00:43:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 11658 2024-04-24T00:43:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=11773], [object_number=32, tag_tid=8, entry_tid=11772], [object_number=35, tag_tid=8, entry_tid=11771], [object_number=34, tag_tid=8, entry_tid=11770]]], state=connected] 2024-04-24T00:43:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=11773], [object_number=32, tag_tid=8, entry_tid=11772], [object_number=35, tag_tid=8, entry_tid=11771], [object_number=34, tag_tid=8, entry_tid=11770]]], state=connected] 2024-04-24T00:43:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=11773' 2024-04-24T00:43:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=11773' 2024-04-24T00:43:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=11773' '!=' 'object_number=34, tag_tid=8, entry_tid=11658' 2024-04-24T00:43:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=11773' = 'object_number=34, tag_tid=8, entry_tid=11658' 2024-04-24T00:43:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=11773' 2024-04-24T00:43:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=11773' 2024-04-24T00:43:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=11773' 2024-04-24T00:43:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11773 2024-04-24T00:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 11773 2024-04-24T00:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=32, tag_tid=8, entry_tid=11888], [object_number=35, tag_tid=8, entry_tid=11887], [object_number=34, tag_tid=8, entry_tid=11886], [object_number=33, tag_tid=8, entry_tid=11885]]], state=connected] 2024-04-24T00:43:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=32, tag_tid=8, entry_tid=11888], [object_number=35, tag_tid=8, entry_tid=11887], [object_number=34, tag_tid=8, entry_tid=11886], [object_number=33, tag_tid=8, entry_tid=11885]]], state=connected] 2024-04-24T00:43:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=32, tag_tid=8, entry_tid=11888' 2024-04-24T00:43:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=32, tag_tid=8, entry_tid=11888' 2024-04-24T00:43:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=32, tag_tid=8, entry_tid=11888' '!=' 'object_number=33, tag_tid=8, entry_tid=11773' 2024-04-24T00:43:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=32, tag_tid=8, entry_tid=11888' = 'object_number=33, tag_tid=8, entry_tid=11773' 2024-04-24T00:43:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=32, tag_tid=8, entry_tid=11888' 2024-04-24T00:43:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=32, tag_tid=8, entry_tid=11888' 2024-04-24T00:43:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=32, tag_tid=8, entry_tid=11888' 2024-04-24T00:43:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11888 2024-04-24T00:43:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 11888 2024-04-24T00:43:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=32, tag_tid=8, entry_tid=12052], [object_number=35, tag_tid=8, entry_tid=12051], [object_number=34, tag_tid=8, entry_tid=12050], [object_number=33, tag_tid=8, entry_tid=12049]]], state=connected] 2024-04-24T00:43:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=32, tag_tid=8, entry_tid=12052], [object_number=35, tag_tid=8, entry_tid=12051], [object_number=34, tag_tid=8, entry_tid=12050], [object_number=33, tag_tid=8, entry_tid=12049]]], state=connected] 2024-04-24T00:43:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=32, tag_tid=8, entry_tid=12052' 2024-04-24T00:43:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=32, tag_tid=8, entry_tid=12052' 2024-04-24T00:43:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=32, tag_tid=8, entry_tid=12052' '!=' 'object_number=32, tag_tid=8, entry_tid=11888' 2024-04-24T00:43:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=32, tag_tid=8, entry_tid=12052' = 'object_number=32, tag_tid=8, entry_tid=11888' 2024-04-24T00:43:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=32, tag_tid=8, entry_tid=12052' 2024-04-24T00:43:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=32, tag_tid=8, entry_tid=12052' 2024-04-24T00:43:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=32, tag_tid=8, entry_tid=12052' 2024-04-24T00:43:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12052 2024-04-24T00:43:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 12052 2024-04-24T00:43:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:10.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=12171], [object_number=34, tag_tid=8, entry_tid=12170], [object_number=33, tag_tid=8, entry_tid=12169], [object_number=32, tag_tid=8, entry_tid=12168]]], state=connected] 2024-04-24T00:43:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=12171], [object_number=34, tag_tid=8, entry_tid=12170], [object_number=33, tag_tid=8, entry_tid=12169], [object_number=32, tag_tid=8, entry_tid=12168]]], state=connected] 2024-04-24T00:43:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=35, tag_tid=8, entry_tid=12171' 2024-04-24T00:43:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=35, tag_tid=8, entry_tid=12171' 2024-04-24T00:43:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=12171' '!=' 'object_number=32, tag_tid=8, entry_tid=12052' 2024-04-24T00:43:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=12171' = 'object_number=32, tag_tid=8, entry_tid=12052' 2024-04-24T00:43:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=35, tag_tid=8, entry_tid=12171' 2024-04-24T00:43:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=12171' 2024-04-24T00:43:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=12171' 2024-04-24T00:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12171 2024-04-24T00:43:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 12171 2024-04-24T00:43:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=12286], [object_number=33, tag_tid=8, entry_tid=12285], [object_number=32, tag_tid=8, entry_tid=12284], [object_number=35, tag_tid=8, entry_tid=12283]]], state=connected] 2024-04-24T00:43:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=34, tag_tid=8, entry_tid=12286], [object_number=33, tag_tid=8, entry_tid=12285], [object_number=32, tag_tid=8, entry_tid=12284], [object_number=35, tag_tid=8, entry_tid=12283]]], state=connected] 2024-04-24T00:43:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=34, tag_tid=8, entry_tid=12286' 2024-04-24T00:43:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=34, tag_tid=8, entry_tid=12286' 2024-04-24T00:43:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=12286' '!=' 'object_number=35, tag_tid=8, entry_tid=12171' 2024-04-24T00:43:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=34, tag_tid=8, entry_tid=12286' = 'object_number=35, tag_tid=8, entry_tid=12171' 2024-04-24T00:43:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=34, tag_tid=8, entry_tid=12286' 2024-04-24T00:43:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=12286' 2024-04-24T00:43:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=34, tag_tid=8, entry_tid=12286' 2024-04-24T00:43:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12286 2024-04-24T00:43:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 12286 2024-04-24T00:43:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:11.681 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.30682118918620505) > weight (0.25) 2024-04-24T00:43:11.681 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6608425032938232) > weight (0.25) 2024-04-24T00:43:11.681 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9469274564141092) > weight (0.25) 2024-04-24T00:43:11.681 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-24T00:43:11.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-24T00:43:11.682 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:11.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T00:43:11.678+0000 7fb8408a3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 29800) UID: 1000 2024-04-24T00:43:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=12399], [object_number=34, tag_tid=8, entry_tid=12398], [object_number=33, tag_tid=8, entry_tid=12397], [object_number=32, tag_tid=8, entry_tid=12396]]], state=connected] 2024-04-24T00:43:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=12399], [object_number=34, tag_tid=8, entry_tid=12398], [object_number=33, tag_tid=8, entry_tid=12397], [object_number=32, tag_tid=8, entry_tid=12396]]], state=connected] 2024-04-24T00:43:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=35, tag_tid=8, entry_tid=12399' 2024-04-24T00:43:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=35, tag_tid=8, entry_tid=12399' 2024-04-24T00:43:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=12399' '!=' 'object_number=34, tag_tid=8, entry_tid=12286' 2024-04-24T00:43:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=12399' = 'object_number=34, tag_tid=8, entry_tid=12286' 2024-04-24T00:43:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=35, tag_tid=8, entry_tid=12399' 2024-04-24T00:43:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=12399' 2024-04-24T00:43:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=12399' 2024-04-24T00:43:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12399 2024-04-24T00:43:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 12399 2024-04-24T00:43:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=12533], [object_number=32, tag_tid=8, entry_tid=12532], [object_number=35, tag_tid=8, entry_tid=12531], [object_number=34, tag_tid=8, entry_tid=12530]]], state=connected] 2024-04-24T00:43:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=12533], [object_number=32, tag_tid=8, entry_tid=12532], [object_number=35, tag_tid=8, entry_tid=12531], [object_number=34, tag_tid=8, entry_tid=12530]]], state=connected] 2024-04-24T00:43:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=12533' 2024-04-24T00:43:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=12533' 2024-04-24T00:43:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=12533' '!=' 'object_number=35, tag_tid=8, entry_tid=12399' 2024-04-24T00:43:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=12533' = 'object_number=35, tag_tid=8, entry_tid=12399' 2024-04-24T00:43:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=12533' 2024-04-24T00:43:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=12533' 2024-04-24T00:43:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=12533' 2024-04-24T00:43:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12533 2024-04-24T00:43:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 12533 2024-04-24T00:43:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=12655], [object_number=34, tag_tid=8, entry_tid=12654], [object_number=33, tag_tid=8, entry_tid=12653], [object_number=32, tag_tid=8, entry_tid=12652]]], state=connected] 2024-04-24T00:43:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=12655], [object_number=34, tag_tid=8, entry_tid=12654], [object_number=33, tag_tid=8, entry_tid=12653], [object_number=32, tag_tid=8, entry_tid=12652]]], state=connected] 2024-04-24T00:43:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=35, tag_tid=8, entry_tid=12655' 2024-04-24T00:43:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=35, tag_tid=8, entry_tid=12655' 2024-04-24T00:43:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=12655' '!=' 'object_number=33, tag_tid=8, entry_tid=12533' 2024-04-24T00:43:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=12655' = 'object_number=33, tag_tid=8, entry_tid=12533' 2024-04-24T00:43:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=35, tag_tid=8, entry_tid=12655' 2024-04-24T00:43:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=12655' 2024-04-24T00:43:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=12655' 2024-04-24T00:43:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12655 2024-04-24T00:43:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 12655 2024-04-24T00:43:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=32, tag_tid=8, entry_tid=12772], [object_number=35, tag_tid=8, entry_tid=12771], [object_number=34, tag_tid=8, entry_tid=12770], [object_number=33, tag_tid=8, entry_tid=12769]]], state=connected] 2024-04-24T00:43:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=32, tag_tid=8, entry_tid=12772], [object_number=35, tag_tid=8, entry_tid=12771], [object_number=34, tag_tid=8, entry_tid=12770], [object_number=33, tag_tid=8, entry_tid=12769]]], state=connected] 2024-04-24T00:43:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=32, tag_tid=8, entry_tid=12772' 2024-04-24T00:43:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=32, tag_tid=8, entry_tid=12772' 2024-04-24T00:43:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=32, tag_tid=8, entry_tid=12772' '!=' 'object_number=35, tag_tid=8, entry_tid=12655' 2024-04-24T00:43:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=32, tag_tid=8, entry_tid=12772' = 'object_number=35, tag_tid=8, entry_tid=12655' 2024-04-24T00:43:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=32, tag_tid=8, entry_tid=12772' 2024-04-24T00:43:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=32, tag_tid=8, entry_tid=12772' 2024-04-24T00:43:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=32, tag_tid=8, entry_tid=12772' 2024-04-24T00:43:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12772 2024-04-24T00:43:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 12772 2024-04-24T00:43:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=12893], [object_number=32, tag_tid=8, entry_tid=12892], [object_number=35, tag_tid=8, entry_tid=12891], [object_number=34, tag_tid=8, entry_tid=12890]]], state=connected] 2024-04-24T00:43:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=12893], [object_number=32, tag_tid=8, entry_tid=12892], [object_number=35, tag_tid=8, entry_tid=12891], [object_number=34, tag_tid=8, entry_tid=12890]]], state=connected] 2024-04-24T00:43:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=12893' 2024-04-24T00:43:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=12893' 2024-04-24T00:43:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=12893' '!=' 'object_number=32, tag_tid=8, entry_tid=12772' 2024-04-24T00:43:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=12893' = 'object_number=32, tag_tid=8, entry_tid=12772' 2024-04-24T00:43:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=12893' 2024-04-24T00:43:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=12893' 2024-04-24T00:43:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=12893' 2024-04-24T00:43:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12893 2024-04-24T00:43:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 12893 2024-04-24T00:43:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13017], [object_number=32, tag_tid=8, entry_tid=13016], [object_number=35, tag_tid=8, entry_tid=13015], [object_number=34, tag_tid=8, entry_tid=13014]]], state=connected] 2024-04-24T00:43:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13017], [object_number=32, tag_tid=8, entry_tid=13016], [object_number=35, tag_tid=8, entry_tid=13015], [object_number=34, tag_tid=8, entry_tid=13014]]], state=connected] 2024-04-24T00:43:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=13017' 2024-04-24T00:43:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=13017' 2024-04-24T00:43:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13017' '!=' 'object_number=33, tag_tid=8, entry_tid=12893' 2024-04-24T00:43:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13017' = 'object_number=33, tag_tid=8, entry_tid=12893' 2024-04-24T00:43:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=13017' 2024-04-24T00:43:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13017' 2024-04-24T00:43:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13017' 2024-04-24T00:43:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13017 2024-04-24T00:43:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 13017 2024-04-24T00:43:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13133], [object_number=32, tag_tid=8, entry_tid=13132], [object_number=35, tag_tid=8, entry_tid=13131], [object_number=34, tag_tid=8, entry_tid=13130]]], state=connected] 2024-04-24T00:43:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13133], [object_number=32, tag_tid=8, entry_tid=13132], [object_number=35, tag_tid=8, entry_tid=13131], [object_number=34, tag_tid=8, entry_tid=13130]]], state=connected] 2024-04-24T00:43:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=13133' 2024-04-24T00:43:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=13133' 2024-04-24T00:43:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13133' '!=' 'object_number=33, tag_tid=8, entry_tid=13017' 2024-04-24T00:43:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13133' = 'object_number=33, tag_tid=8, entry_tid=13017' 2024-04-24T00:43:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=13133' 2024-04-24T00:43:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13133' 2024-04-24T00:43:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13133' 2024-04-24T00:43:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13133 2024-04-24T00:43:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 13133 2024-04-24T00:43:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=32, tag_tid=8, entry_tid=13292], [object_number=35, tag_tid=8, entry_tid=13291], [object_number=34, tag_tid=8, entry_tid=13290], [object_number=33, tag_tid=8, entry_tid=13289]]], state=connected] 2024-04-24T00:43:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=32, tag_tid=8, entry_tid=13292], [object_number=35, tag_tid=8, entry_tid=13291], [object_number=34, tag_tid=8, entry_tid=13290], [object_number=33, tag_tid=8, entry_tid=13289]]], state=connected] 2024-04-24T00:43:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=32, tag_tid=8, entry_tid=13292' 2024-04-24T00:43:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=32, tag_tid=8, entry_tid=13292' 2024-04-24T00:43:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=32, tag_tid=8, entry_tid=13292' '!=' 'object_number=33, tag_tid=8, entry_tid=13133' 2024-04-24T00:43:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=32, tag_tid=8, entry_tid=13292' = 'object_number=33, tag_tid=8, entry_tid=13133' 2024-04-24T00:43:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=32, tag_tid=8, entry_tid=13292' 2024-04-24T00:43:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=32, tag_tid=8, entry_tid=13292' 2024-04-24T00:43:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=32, tag_tid=8, entry_tid=13292' 2024-04-24T00:43:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13292 2024-04-24T00:43:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 13292 2024-04-24T00:43:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=13419], [object_number=34, tag_tid=8, entry_tid=13418], [object_number=33, tag_tid=8, entry_tid=13417], [object_number=32, tag_tid=8, entry_tid=13416]]], state=connected] 2024-04-24T00:43:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=13419], [object_number=34, tag_tid=8, entry_tid=13418], [object_number=33, tag_tid=8, entry_tid=13417], [object_number=32, tag_tid=8, entry_tid=13416]]], state=connected] 2024-04-24T00:43:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=35, tag_tid=8, entry_tid=13419' 2024-04-24T00:43:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=35, tag_tid=8, entry_tid=13419' 2024-04-24T00:43:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=13419' '!=' 'object_number=32, tag_tid=8, entry_tid=13292' 2024-04-24T00:43:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=13419' = 'object_number=32, tag_tid=8, entry_tid=13292' 2024-04-24T00:43:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=35, tag_tid=8, entry_tid=13419' 2024-04-24T00:43:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=13419' 2024-04-24T00:43:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=13419' 2024-04-24T00:43:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13419 2024-04-24T00:43:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 13419 2024-04-24T00:43:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13545], [object_number=32, tag_tid=8, entry_tid=13544], [object_number=35, tag_tid=8, entry_tid=13543], [object_number=34, tag_tid=8, entry_tid=13542]]], state=connected] 2024-04-24T00:43:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13545], [object_number=32, tag_tid=8, entry_tid=13544], [object_number=35, tag_tid=8, entry_tid=13543], [object_number=34, tag_tid=8, entry_tid=13542]]], state=connected] 2024-04-24T00:43:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=13545' 2024-04-24T00:43:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=13545' 2024-04-24T00:43:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13545' '!=' 'object_number=35, tag_tid=8, entry_tid=13419' 2024-04-24T00:43:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13545' = 'object_number=35, tag_tid=8, entry_tid=13419' 2024-04-24T00:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=13545' 2024-04-24T00:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13545' 2024-04-24T00:43:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13545' 2024-04-24T00:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13545 2024-04-24T00:43:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 13545 2024-04-24T00:43:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:18.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:18.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:18.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13669], [object_number=32, tag_tid=8, entry_tid=13668], [object_number=35, tag_tid=8, entry_tid=13667], [object_number=34, tag_tid=8, entry_tid=13666]]], state=connected] 2024-04-24T00:43:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13669], [object_number=32, tag_tid=8, entry_tid=13668], [object_number=35, tag_tid=8, entry_tid=13667], [object_number=34, tag_tid=8, entry_tid=13666]]], state=connected] 2024-04-24T00:43:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=13669' 2024-04-24T00:43:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=13669' 2024-04-24T00:43:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13669' '!=' 'object_number=33, tag_tid=8, entry_tid=13545' 2024-04-24T00:43:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13669' = 'object_number=33, tag_tid=8, entry_tid=13545' 2024-04-24T00:43:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=13669' 2024-04-24T00:43:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13669' 2024-04-24T00:43:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13669' 2024-04-24T00:43:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13669 2024-04-24T00:43:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 13669 2024-04-24T00:43:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:18.682 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-24T00:43:18.682 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:43:18.683 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-24T00:43:18.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-24T00:43:18.683 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-24T00:43:18.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-24T00:43:18.686 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-04-24T00:43:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13797], [object_number=32, tag_tid=8, entry_tid=13796], [object_number=35, tag_tid=8, entry_tid=13795], [object_number=34, tag_tid=8, entry_tid=13794]]], state=connected] 2024-04-24T00:43:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=13797], [object_number=32, tag_tid=8, entry_tid=13796], [object_number=35, tag_tid=8, entry_tid=13795], [object_number=34, tag_tid=8, entry_tid=13794]]], state=connected] 2024-04-24T00:43:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=13797' 2024-04-24T00:43:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=13797' 2024-04-24T00:43:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13797' '!=' 'object_number=33, tag_tid=8, entry_tid=13669' 2024-04-24T00:43:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=13797' = 'object_number=33, tag_tid=8, entry_tid=13669' 2024-04-24T00:43:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=13797' 2024-04-24T00:43:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13797' 2024-04-24T00:43:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=13797' 2024-04-24T00:43:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13797 2024-04-24T00:43:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 13797 2024-04-24T00:43:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=13915], [object_number=34, tag_tid=8, entry_tid=13914], [object_number=33, tag_tid=8, entry_tid=13913], [object_number=32, tag_tid=8, entry_tid=13912]]], state=connected] 2024-04-24T00:43:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=35, tag_tid=8, entry_tid=13915], [object_number=34, tag_tid=8, entry_tid=13914], [object_number=33, tag_tid=8, entry_tid=13913], [object_number=32, tag_tid=8, entry_tid=13912]]], state=connected] 2024-04-24T00:43:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=35, tag_tid=8, entry_tid=13915' 2024-04-24T00:43:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=35, tag_tid=8, entry_tid=13915' 2024-04-24T00:43:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=13915' '!=' 'object_number=33, tag_tid=8, entry_tid=13797' 2024-04-24T00:43:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=35, tag_tid=8, entry_tid=13915' = 'object_number=33, tag_tid=8, entry_tid=13797' 2024-04-24T00:43:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=35, tag_tid=8, entry_tid=13915' 2024-04-24T00:43:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=13915' 2024-04-24T00:43:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=35, tag_tid=8, entry_tid=13915' 2024-04-24T00:43:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13915 2024-04-24T00:43:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 13915 2024-04-24T00:43:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=14033], [object_number=32, tag_tid=8, entry_tid=14032], [object_number=35, tag_tid=8, entry_tid=14031], [object_number=34, tag_tid=8, entry_tid=14030]]], state=connected] 2024-04-24T00:43:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=14033], [object_number=32, tag_tid=8, entry_tid=14032], [object_number=35, tag_tid=8, entry_tid=14031], [object_number=34, tag_tid=8, entry_tid=14030]]], state=connected] 2024-04-24T00:43:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=14033' 2024-04-24T00:43:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=14033' 2024-04-24T00:43:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=14033' '!=' 'object_number=35, tag_tid=8, entry_tid=13915' 2024-04-24T00:43:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=14033' = 'object_number=35, tag_tid=8, entry_tid=13915' 2024-04-24T00:43:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=14033' 2024-04-24T00:43:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=14033' 2024-04-24T00:43:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=14033' 2024-04-24T00:43:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14033 2024-04-24T00:43:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 14033 2024-04-24T00:43:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=14145], [object_number=32, tag_tid=8, entry_tid=14144], [object_number=35, tag_tid=8, entry_tid=14143], [object_number=34, tag_tid=8, entry_tid=14142]]], state=connected] 2024-04-24T00:43:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 8 2024-04-24T00:43:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=33, tag_tid=8, entry_tid=14145], [object_number=32, tag_tid=8, entry_tid=14144], [object_number=35, tag_tid=8, entry_tid=14143], [object_number=34, tag_tid=8, entry_tid=14142]]], state=connected] 2024-04-24T00:43:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=33, tag_tid=8, entry_tid=14145' 2024-04-24T00:43:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=33, tag_tid=8, entry_tid=14145' 2024-04-24T00:43:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=14145' '!=' 'object_number=33, tag_tid=8, entry_tid=14033' 2024-04-24T00:43:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=33, tag_tid=8, entry_tid=14145' = 'object_number=33, tag_tid=8, entry_tid=14033' 2024-04-24T00:43:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=33, tag_tid=8, entry_tid=14145' 2024-04-24T00:43:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=14145' 2024-04-24T00:43:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=33, tag_tid=8, entry_tid=14145' 2024-04-24T00:43:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14145 2024-04-24T00:43:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 14145 2024-04-24T00:43:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=14238], [object_number=37, tag_tid=8, entry_tid=14237], [object_number=36, tag_tid=8, entry_tid=14236], [object_number=39, tag_tid=8, entry_tid=14235]]], state=connected] 2024-04-24T00:43:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=14238], [object_number=37, tag_tid=8, entry_tid=14237], [object_number=36, tag_tid=8, entry_tid=14236], [object_number=39, tag_tid=8, entry_tid=14235]]], state=connected] 2024-04-24T00:43:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=14238' 2024-04-24T00:43:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=14238' 2024-04-24T00:43:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=14238' '!=' 'object_number=33, tag_tid=8, entry_tid=14145' 2024-04-24T00:43:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=14238' = 'object_number=33, tag_tid=8, entry_tid=14145' 2024-04-24T00:43:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=14238' 2024-04-24T00:43:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=14238' 2024-04-24T00:43:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=14238' 2024-04-24T00:43:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14238 2024-04-24T00:43:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 14238 2024-04-24T00:43:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=14365], [object_number=36, tag_tid=8, entry_tid=14364], [object_number=39, tag_tid=8, entry_tid=14363], [object_number=38, tag_tid=8, entry_tid=14362]]], state=connected] 2024-04-24T00:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=14365], [object_number=36, tag_tid=8, entry_tid=14364], [object_number=39, tag_tid=8, entry_tid=14363], [object_number=38, tag_tid=8, entry_tid=14362]]], state=connected] 2024-04-24T00:43:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=37, tag_tid=8, entry_tid=14365' 2024-04-24T00:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=37, tag_tid=8, entry_tid=14365' 2024-04-24T00:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=14365' '!=' 'object_number=38, tag_tid=8, entry_tid=14238' 2024-04-24T00:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=14365' = 'object_number=38, tag_tid=8, entry_tid=14238' 2024-04-24T00:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=37, tag_tid=8, entry_tid=14365' 2024-04-24T00:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=14365' 2024-04-24T00:43:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=14365' 2024-04-24T00:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14365 2024-04-24T00:43:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 14365 2024-04-24T00:43:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=14499], [object_number=38, tag_tid=8, entry_tid=14498], [object_number=37, tag_tid=8, entry_tid=14497], [object_number=36, tag_tid=8, entry_tid=14496]]], state=connected] 2024-04-24T00:43:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=14499], [object_number=38, tag_tid=8, entry_tid=14498], [object_number=37, tag_tid=8, entry_tid=14497], [object_number=36, tag_tid=8, entry_tid=14496]]], state=connected] 2024-04-24T00:43:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=14499' 2024-04-24T00:43:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=14499' 2024-04-24T00:43:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=14499' '!=' 'object_number=37, tag_tid=8, entry_tid=14365' 2024-04-24T00:43:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=14499' = 'object_number=37, tag_tid=8, entry_tid=14365' 2024-04-24T00:43:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=14499' 2024-04-24T00:43:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=14499' 2024-04-24T00:43:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=14499' 2024-04-24T00:43:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14499 2024-04-24T00:43:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 14499 2024-04-24T00:43:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=14683], [object_number=38, tag_tid=8, entry_tid=14682], [object_number=37, tag_tid=8, entry_tid=14681], [object_number=36, tag_tid=8, entry_tid=14680]]], state=connected] 2024-04-24T00:43:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=14683], [object_number=38, tag_tid=8, entry_tid=14682], [object_number=37, tag_tid=8, entry_tid=14681], [object_number=36, tag_tid=8, entry_tid=14680]]], state=connected] 2024-04-24T00:43:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=14683' 2024-04-24T00:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=14683' 2024-04-24T00:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=14683' '!=' 'object_number=39, tag_tid=8, entry_tid=14499' 2024-04-24T00:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=14683' = 'object_number=39, tag_tid=8, entry_tid=14499' 2024-04-24T00:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=14683' 2024-04-24T00:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=14683' 2024-04-24T00:43:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=14683' 2024-04-24T00:43:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14683 2024-04-24T00:43:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 14683 2024-04-24T00:43:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=14815], [object_number=38, tag_tid=8, entry_tid=14814], [object_number=37, tag_tid=8, entry_tid=14813], [object_number=36, tag_tid=8, entry_tid=14812]]], state=connected] 2024-04-24T00:43:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=14815], [object_number=38, tag_tid=8, entry_tid=14814], [object_number=37, tag_tid=8, entry_tid=14813], [object_number=36, tag_tid=8, entry_tid=14812]]], state=connected] 2024-04-24T00:43:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=14815' 2024-04-24T00:43:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=14815' 2024-04-24T00:43:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=14815' '!=' 'object_number=39, tag_tid=8, entry_tid=14683' 2024-04-24T00:43:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=14815' = 'object_number=39, tag_tid=8, entry_tid=14683' 2024-04-24T00:43:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=14815' 2024-04-24T00:43:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=14815' 2024-04-24T00:43:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=14815' 2024-04-24T00:43:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14815 2024-04-24T00:43:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 14815 2024-04-24T00:43:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=14938], [object_number=37, tag_tid=8, entry_tid=14937], [object_number=36, tag_tid=8, entry_tid=14936], [object_number=39, tag_tid=8, entry_tid=14935]]], state=connected] 2024-04-24T00:43:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.676 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T00:43:24.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T00:43:24.677 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-24T00:43:24.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T00:43:24.674+0000 7fa1853b1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36558) UID: 1000 2024-04-24T00:43:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=14938], [object_number=37, tag_tid=8, entry_tid=14937], [object_number=36, tag_tid=8, entry_tid=14936], [object_number=39, tag_tid=8, entry_tid=14935]]], state=connected] 2024-04-24T00:43:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=14938' 2024-04-24T00:43:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=14938' 2024-04-24T00:43:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=14938' '!=' 'object_number=39, tag_tid=8, entry_tid=14815' 2024-04-24T00:43:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=14938' = 'object_number=39, tag_tid=8, entry_tid=14815' 2024-04-24T00:43:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=14938' 2024-04-24T00:43:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=14938' 2024-04-24T00:43:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=14938' 2024-04-24T00:43:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14938 2024-04-24T00:43:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 14938 2024-04-24T00:43:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:43:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15058], [object_number=37, tag_tid=8, entry_tid=15057], [object_number=36, tag_tid=8, entry_tid=15056], [object_number=39, tag_tid=8, entry_tid=15055]]], state=connected] 2024-04-24T00:43:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15058], [object_number=37, tag_tid=8, entry_tid=15057], [object_number=36, tag_tid=8, entry_tid=15056], [object_number=39, tag_tid=8, entry_tid=15055]]], state=connected] 2024-04-24T00:43:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=15058' 2024-04-24T00:43:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=15058' 2024-04-24T00:43:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15058' '!=' 'object_number=38, tag_tid=8, entry_tid=14938' 2024-04-24T00:43:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15058' = 'object_number=38, tag_tid=8, entry_tid=14938' 2024-04-24T00:43:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=15058' 2024-04-24T00:43:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15058' 2024-04-24T00:43:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15058' 2024-04-24T00:43:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15058 2024-04-24T00:43:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 15058 2024-04-24T00:43:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15170], [object_number=37, tag_tid=8, entry_tid=15169], [object_number=36, tag_tid=8, entry_tid=15168], [object_number=39, tag_tid=8, entry_tid=15167]]], state=connected] 2024-04-24T00:43:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:25.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:25.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15170], [object_number=37, tag_tid=8, entry_tid=15169], [object_number=36, tag_tid=8, entry_tid=15168], [object_number=39, tag_tid=8, entry_tid=15167]]], state=connected] 2024-04-24T00:43:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=15170' 2024-04-24T00:43:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=15170' 2024-04-24T00:43:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15170' '!=' 'object_number=38, tag_tid=8, entry_tid=15058' 2024-04-24T00:43:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15170' = 'object_number=38, tag_tid=8, entry_tid=15058' 2024-04-24T00:43:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=15170' 2024-04-24T00:43:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15170' 2024-04-24T00:43:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15170' 2024-04-24T00:43:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15170 2024-04-24T00:43:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 15170 2024-04-24T00:43:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:43:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=15279], [object_number=38, tag_tid=8, entry_tid=15278], [object_number=37, tag_tid=8, entry_tid=15277], [object_number=36, tag_tid=8, entry_tid=15276]]], state=connected] 2024-04-24T00:43:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=15279], [object_number=38, tag_tid=8, entry_tid=15278], [object_number=37, tag_tid=8, entry_tid=15277], [object_number=36, tag_tid=8, entry_tid=15276]]], state=connected] 2024-04-24T00:43:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=15279' 2024-04-24T00:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=15279' 2024-04-24T00:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=15279' '!=' 'object_number=38, tag_tid=8, entry_tid=15170' 2024-04-24T00:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=15279' = 'object_number=38, tag_tid=8, entry_tid=15170' 2024-04-24T00:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=15279' 2024-04-24T00:43:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=15279' 2024-04-24T00:43:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=15279' 2024-04-24T00:43:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15279 2024-04-24T00:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 15279 2024-04-24T00:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:43:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15398], [object_number=37, tag_tid=8, entry_tid=15397], [object_number=36, tag_tid=8, entry_tid=15396], [object_number=39, tag_tid=8, entry_tid=15395]]], state=connected] 2024-04-24T00:43:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15398], [object_number=37, tag_tid=8, entry_tid=15397], [object_number=36, tag_tid=8, entry_tid=15396], [object_number=39, tag_tid=8, entry_tid=15395]]], state=connected] 2024-04-24T00:43:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=15398' 2024-04-24T00:43:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=15398' 2024-04-24T00:43:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15398' '!=' 'object_number=39, tag_tid=8, entry_tid=15279' 2024-04-24T00:43:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15398' = 'object_number=39, tag_tid=8, entry_tid=15279' 2024-04-24T00:43:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=15398' 2024-04-24T00:43:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15398' 2024-04-24T00:43:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15398' 2024-04-24T00:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15398 2024-04-24T00:43:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 15398 2024-04-24T00:43:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15506], [object_number=37, tag_tid=8, entry_tid=15505], [object_number=36, tag_tid=8, entry_tid=15504], [object_number=39, tag_tid=8, entry_tid=15503]]], state=connected] 2024-04-24T00:43:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15506], [object_number=37, tag_tid=8, entry_tid=15505], [object_number=36, tag_tid=8, entry_tid=15504], [object_number=39, tag_tid=8, entry_tid=15503]]], state=connected] 2024-04-24T00:43:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=15506' 2024-04-24T00:43:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=15506' 2024-04-24T00:43:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15506' '!=' 'object_number=38, tag_tid=8, entry_tid=15398' 2024-04-24T00:43:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15506' = 'object_number=38, tag_tid=8, entry_tid=15398' 2024-04-24T00:43:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=15506' 2024-04-24T00:43:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15506' 2024-04-24T00:43:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15506' 2024-04-24T00:43:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15506 2024-04-24T00:43:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 15506 2024-04-24T00:43:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:43:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=15620], [object_number=39, tag_tid=8, entry_tid=15619], [object_number=38, tag_tid=8, entry_tid=15618], [object_number=37, tag_tid=8, entry_tid=15617]]], state=connected] 2024-04-24T00:43:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=15620], [object_number=39, tag_tid=8, entry_tid=15619], [object_number=38, tag_tid=8, entry_tid=15618], [object_number=37, tag_tid=8, entry_tid=15617]]], state=connected] 2024-04-24T00:43:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=36, tag_tid=8, entry_tid=15620' 2024-04-24T00:43:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=36, tag_tid=8, entry_tid=15620' 2024-04-24T00:43:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=15620' '!=' 'object_number=38, tag_tid=8, entry_tid=15506' 2024-04-24T00:43:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=15620' = 'object_number=38, tag_tid=8, entry_tid=15506' 2024-04-24T00:43:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=36, tag_tid=8, entry_tid=15620' 2024-04-24T00:43:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=15620' 2024-04-24T00:43:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=15620' 2024-04-24T00:43:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15620 2024-04-24T00:43:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 15620 2024-04-24T00:43:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:28.678 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T00:43:28.678 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:43:28.678 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T00:43:28.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T00:43:28.678 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T00:43:28.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T00:43:28.681 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-04-24T00:43:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:43:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15734], [object_number=37, tag_tid=8, entry_tid=15733], [object_number=36, tag_tid=8, entry_tid=15732], [object_number=39, tag_tid=8, entry_tid=15731]]], state=connected] 2024-04-24T00:43:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=15734], [object_number=37, tag_tid=8, entry_tid=15733], [object_number=36, tag_tid=8, entry_tid=15732], [object_number=39, tag_tid=8, entry_tid=15731]]], state=connected] 2024-04-24T00:43:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=15734' 2024-04-24T00:43:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=15734' 2024-04-24T00:43:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15734' '!=' 'object_number=36, tag_tid=8, entry_tid=15620' 2024-04-24T00:43:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=15734' = 'object_number=36, tag_tid=8, entry_tid=15620' 2024-04-24T00:43:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=15734' 2024-04-24T00:43:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15734' 2024-04-24T00:43:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=15734' 2024-04-24T00:43:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15734 2024-04-24T00:43:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 15734 2024-04-24T00:43:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=15852], [object_number=39, tag_tid=8, entry_tid=15851], [object_number=38, tag_tid=8, entry_tid=15850], [object_number=37, tag_tid=8, entry_tid=15849]]], state=connected] 2024-04-24T00:43:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=15852], [object_number=39, tag_tid=8, entry_tid=15851], [object_number=38, tag_tid=8, entry_tid=15850], [object_number=37, tag_tid=8, entry_tid=15849]]], state=connected] 2024-04-24T00:43:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=36, tag_tid=8, entry_tid=15852' 2024-04-24T00:43:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=36, tag_tid=8, entry_tid=15852' 2024-04-24T00:43:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=15852' '!=' 'object_number=38, tag_tid=8, entry_tid=15734' 2024-04-24T00:43:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=15852' = 'object_number=38, tag_tid=8, entry_tid=15734' 2024-04-24T00:43:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=36, tag_tid=8, entry_tid=15852' 2024-04-24T00:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=15852' 2024-04-24T00:43:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=15852' 2024-04-24T00:43:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15852 2024-04-24T00:43:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 15852 2024-04-24T00:43:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=16005], [object_number=36, tag_tid=8, entry_tid=16004], [object_number=39, tag_tid=8, entry_tid=16003], [object_number=38, tag_tid=8, entry_tid=16002]]], state=connected] 2024-04-24T00:43:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=16005], [object_number=36, tag_tid=8, entry_tid=16004], [object_number=39, tag_tid=8, entry_tid=16003], [object_number=38, tag_tid=8, entry_tid=16002]]], state=connected] 2024-04-24T00:43:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=37, tag_tid=8, entry_tid=16005' 2024-04-24T00:43:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=37, tag_tid=8, entry_tid=16005' 2024-04-24T00:43:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=16005' '!=' 'object_number=36, tag_tid=8, entry_tid=15852' 2024-04-24T00:43:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=16005' = 'object_number=36, tag_tid=8, entry_tid=15852' 2024-04-24T00:43:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=37, tag_tid=8, entry_tid=16005' 2024-04-24T00:43:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=16005' 2024-04-24T00:43:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=16005' 2024-04-24T00:43:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16005 2024-04-24T00:43:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 16005 2024-04-24T00:43:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=16117], [object_number=36, tag_tid=8, entry_tid=16116], [object_number=39, tag_tid=8, entry_tid=16115], [object_number=38, tag_tid=8, entry_tid=16114]]], state=connected] 2024-04-24T00:43:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=16117], [object_number=36, tag_tid=8, entry_tid=16116], [object_number=39, tag_tid=8, entry_tid=16115], [object_number=38, tag_tid=8, entry_tid=16114]]], state=connected] 2024-04-24T00:43:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=37, tag_tid=8, entry_tid=16117' 2024-04-24T00:43:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=37, tag_tid=8, entry_tid=16117' 2024-04-24T00:43:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=16117' '!=' 'object_number=37, tag_tid=8, entry_tid=16005' 2024-04-24T00:43:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=16117' = 'object_number=37, tag_tid=8, entry_tid=16005' 2024-04-24T00:43:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=37, tag_tid=8, entry_tid=16117' 2024-04-24T00:43:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=16117' 2024-04-24T00:43:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=16117' 2024-04-24T00:43:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16117 2024-04-24T00:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 16117 2024-04-24T00:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:31.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:31.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:31.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=16240], [object_number=39, tag_tid=8, entry_tid=16239], [object_number=38, tag_tid=8, entry_tid=16238], [object_number=37, tag_tid=8, entry_tid=16237]]], state=connected] 2024-04-24T00:43:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=16240], [object_number=39, tag_tid=8, entry_tid=16239], [object_number=38, tag_tid=8, entry_tid=16238], [object_number=37, tag_tid=8, entry_tid=16237]]], state=connected] 2024-04-24T00:43:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=36, tag_tid=8, entry_tid=16240' 2024-04-24T00:43:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=36, tag_tid=8, entry_tid=16240' 2024-04-24T00:43:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=16240' '!=' 'object_number=37, tag_tid=8, entry_tid=16117' 2024-04-24T00:43:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=16240' = 'object_number=37, tag_tid=8, entry_tid=16117' 2024-04-24T00:43:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=36, tag_tid=8, entry_tid=16240' 2024-04-24T00:43:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=16240' 2024-04-24T00:43:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=16240' 2024-04-24T00:43:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16240 2024-04-24T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 16240 2024-04-24T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=16375], [object_number=38, tag_tid=8, entry_tid=16374], [object_number=37, tag_tid=8, entry_tid=16373], [object_number=36, tag_tid=8, entry_tid=16372]]], state=connected] 2024-04-24T00:43:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=16375], [object_number=38, tag_tid=8, entry_tid=16374], [object_number=37, tag_tid=8, entry_tid=16373], [object_number=36, tag_tid=8, entry_tid=16372]]], state=connected] 2024-04-24T00:43:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=16375' 2024-04-24T00:43:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=16375' 2024-04-24T00:43:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=16375' '!=' 'object_number=36, tag_tid=8, entry_tid=16240' 2024-04-24T00:43:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=16375' = 'object_number=36, tag_tid=8, entry_tid=16240' 2024-04-24T00:43:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=16375' 2024-04-24T00:43:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=16375' 2024-04-24T00:43:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=16375' 2024-04-24T00:43:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16375 2024-04-24T00:43:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 16375 2024-04-24T00:43:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=16526], [object_number=37, tag_tid=8, entry_tid=16525], [object_number=36, tag_tid=8, entry_tid=16524], [object_number=39, tag_tid=8, entry_tid=16523]]], state=connected] 2024-04-24T00:43:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=16526], [object_number=37, tag_tid=8, entry_tid=16525], [object_number=36, tag_tid=8, entry_tid=16524], [object_number=39, tag_tid=8, entry_tid=16523]]], state=connected] 2024-04-24T00:43:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=16526' 2024-04-24T00:43:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=16526' 2024-04-24T00:43:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=16526' '!=' 'object_number=39, tag_tid=8, entry_tid=16375' 2024-04-24T00:43:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=16526' = 'object_number=39, tag_tid=8, entry_tid=16375' 2024-04-24T00:43:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=16526' 2024-04-24T00:43:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=16526' 2024-04-24T00:43:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=16526' 2024-04-24T00:43:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16526 2024-04-24T00:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 16526 2024-04-24T00:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=16638], [object_number=37, tag_tid=8, entry_tid=16637], [object_number=36, tag_tid=8, entry_tid=16636], [object_number=39, tag_tid=8, entry_tid=16635]]], state=connected] 2024-04-24T00:43:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=16638], [object_number=37, tag_tid=8, entry_tid=16637], [object_number=36, tag_tid=8, entry_tid=16636], [object_number=39, tag_tid=8, entry_tid=16635]]], state=connected] 2024-04-24T00:43:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=16638' 2024-04-24T00:43:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=16638' 2024-04-24T00:43:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=16638' '!=' 'object_number=38, tag_tid=8, entry_tid=16526' 2024-04-24T00:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=16638' = 'object_number=38, tag_tid=8, entry_tid=16526' 2024-04-24T00:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=16638' 2024-04-24T00:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=16638' 2024-04-24T00:43:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=16638' 2024-04-24T00:43:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16638 2024-04-24T00:43:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 16638 2024-04-24T00:43:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=16759], [object_number=38, tag_tid=8, entry_tid=16758], [object_number=37, tag_tid=8, entry_tid=16757], [object_number=36, tag_tid=8, entry_tid=16756]]], state=connected] 2024-04-24T00:43:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=16759], [object_number=38, tag_tid=8, entry_tid=16758], [object_number=37, tag_tid=8, entry_tid=16757], [object_number=36, tag_tid=8, entry_tid=16756]]], state=connected] 2024-04-24T00:43:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=16759' 2024-04-24T00:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=16759' 2024-04-24T00:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=16759' '!=' 'object_number=38, tag_tid=8, entry_tid=16638' 2024-04-24T00:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=16759' = 'object_number=38, tag_tid=8, entry_tid=16638' 2024-04-24T00:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=16759' 2024-04-24T00:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=16759' 2024-04-24T00:43:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=16759' 2024-04-24T00:43:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16759 2024-04-24T00:43:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 16759 2024-04-24T00:43:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=16879], [object_number=38, tag_tid=8, entry_tid=16878], [object_number=37, tag_tid=8, entry_tid=16877], [object_number=36, tag_tid=8, entry_tid=16876]]], state=connected] 2024-04-24T00:43:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=16879], [object_number=38, tag_tid=8, entry_tid=16878], [object_number=37, tag_tid=8, entry_tid=16877], [object_number=36, tag_tid=8, entry_tid=16876]]], state=connected] 2024-04-24T00:43:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=16879' 2024-04-24T00:43:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=16879' 2024-04-24T00:43:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=16879' '!=' 'object_number=39, tag_tid=8, entry_tid=16759' 2024-04-24T00:43:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=16879' = 'object_number=39, tag_tid=8, entry_tid=16759' 2024-04-24T00:43:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=16879' 2024-04-24T00:43:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=16879' 2024-04-24T00:43:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=16879' 2024-04-24T00:43:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16879 2024-04-24T00:43:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 16879 2024-04-24T00:43:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=17003], [object_number=38, tag_tid=8, entry_tid=17002], [object_number=37, tag_tid=8, entry_tid=17001], [object_number=36, tag_tid=8, entry_tid=17000]]], state=connected] 2024-04-24T00:43:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=17003], [object_number=38, tag_tid=8, entry_tid=17002], [object_number=37, tag_tid=8, entry_tid=17001], [object_number=36, tag_tid=8, entry_tid=17000]]], state=connected] 2024-04-24T00:43:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=17003' 2024-04-24T00:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=17003' 2024-04-24T00:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=17003' '!=' 'object_number=39, tag_tid=8, entry_tid=16879' 2024-04-24T00:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=17003' = 'object_number=39, tag_tid=8, entry_tid=16879' 2024-04-24T00:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=17003' 2024-04-24T00:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=17003' 2024-04-24T00:43:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=17003' 2024-04-24T00:43:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17003 2024-04-24T00:43:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 17003 2024-04-24T00:43:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=17123], [object_number=38, tag_tid=8, entry_tid=17122], [object_number=37, tag_tid=8, entry_tid=17121], [object_number=36, tag_tid=8, entry_tid=17120]]], state=connected] 2024-04-24T00:43:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=17123], [object_number=38, tag_tid=8, entry_tid=17122], [object_number=37, tag_tid=8, entry_tid=17121], [object_number=36, tag_tid=8, entry_tid=17120]]], state=connected] 2024-04-24T00:43:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=17123' 2024-04-24T00:43:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=17123' 2024-04-24T00:43:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=17123' '!=' 'object_number=39, tag_tid=8, entry_tid=17003' 2024-04-24T00:43:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=17123' = 'object_number=39, tag_tid=8, entry_tid=17003' 2024-04-24T00:43:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=17123' 2024-04-24T00:43:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=17123' 2024-04-24T00:43:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=17123' 2024-04-24T00:43:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17123 2024-04-24T00:43:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 17123 2024-04-24T00:43:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=17226], [object_number=37, tag_tid=8, entry_tid=17225], [object_number=36, tag_tid=8, entry_tid=17224], [object_number=39, tag_tid=8, entry_tid=17223]]], state=connected] 2024-04-24T00:43:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=17226], [object_number=37, tag_tid=8, entry_tid=17225], [object_number=36, tag_tid=8, entry_tid=17224], [object_number=39, tag_tid=8, entry_tid=17223]]], state=connected] 2024-04-24T00:43:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=17226' 2024-04-24T00:43:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=17226' 2024-04-24T00:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=17226' '!=' 'object_number=39, tag_tid=8, entry_tid=17123' 2024-04-24T00:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=17226' = 'object_number=39, tag_tid=8, entry_tid=17123' 2024-04-24T00:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=17226' 2024-04-24T00:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=17226' 2024-04-24T00:43:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=17226' 2024-04-24T00:43:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17226 2024-04-24T00:43:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 17226 2024-04-24T00:43:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=17334], [object_number=37, tag_tid=8, entry_tid=17333], [object_number=36, tag_tid=8, entry_tid=17332], [object_number=39, tag_tid=8, entry_tid=17331]]], state=connected] 2024-04-24T00:43:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=17334], [object_number=37, tag_tid=8, entry_tid=17333], [object_number=36, tag_tid=8, entry_tid=17332], [object_number=39, tag_tid=8, entry_tid=17331]]], state=connected] 2024-04-24T00:43:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=17334' 2024-04-24T00:43:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=17334' 2024-04-24T00:43:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=17334' '!=' 'object_number=38, tag_tid=8, entry_tid=17226' 2024-04-24T00:43:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=17334' = 'object_number=38, tag_tid=8, entry_tid=17226' 2024-04-24T00:43:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=17334' 2024-04-24T00:43:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=17334' 2024-04-24T00:43:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=17334' 2024-04-24T00:43:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17334 2024-04-24T00:43:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 17334 2024-04-24T00:43:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=17440], [object_number=39, tag_tid=8, entry_tid=17439], [object_number=38, tag_tid=8, entry_tid=17438], [object_number=37, tag_tid=8, entry_tid=17437]]], state=connected] 2024-04-24T00:43:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=17440], [object_number=39, tag_tid=8, entry_tid=17439], [object_number=38, tag_tid=8, entry_tid=17438], [object_number=37, tag_tid=8, entry_tid=17437]]], state=connected] 2024-04-24T00:43:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=36, tag_tid=8, entry_tid=17440' 2024-04-24T00:43:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=36, tag_tid=8, entry_tid=17440' 2024-04-24T00:43:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=17440' '!=' 'object_number=38, tag_tid=8, entry_tid=17334' 2024-04-24T00:43:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=17440' = 'object_number=38, tag_tid=8, entry_tid=17334' 2024-04-24T00:43:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=36, tag_tid=8, entry_tid=17440' 2024-04-24T00:43:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=17440' 2024-04-24T00:43:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=17440' 2024-04-24T00:43:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17440 2024-04-24T00:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 17440 2024-04-24T00:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:38.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:38.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=17562], [object_number=37, tag_tid=8, entry_tid=17561], [object_number=36, tag_tid=8, entry_tid=17560], [object_number=39, tag_tid=8, entry_tid=17559]]], state=connected] 2024-04-24T00:43:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=38, tag_tid=8, entry_tid=17562], [object_number=37, tag_tid=8, entry_tid=17561], [object_number=36, tag_tid=8, entry_tid=17560], [object_number=39, tag_tid=8, entry_tid=17559]]], state=connected] 2024-04-24T00:43:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=38, tag_tid=8, entry_tid=17562' 2024-04-24T00:43:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=38, tag_tid=8, entry_tid=17562' 2024-04-24T00:43:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=17562' '!=' 'object_number=36, tag_tid=8, entry_tid=17440' 2024-04-24T00:43:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=38, tag_tid=8, entry_tid=17562' = 'object_number=36, tag_tid=8, entry_tid=17440' 2024-04-24T00:43:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=38, tag_tid=8, entry_tid=17562' 2024-04-24T00:43:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=17562' 2024-04-24T00:43:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=38, tag_tid=8, entry_tid=17562' 2024-04-24T00:43:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17562 2024-04-24T00:43:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 17562 2024-04-24T00:43:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=17681], [object_number=36, tag_tid=8, entry_tid=17680], [object_number=39, tag_tid=8, entry_tid=17679], [object_number=38, tag_tid=8, entry_tid=17678]]], state=connected] 2024-04-24T00:43:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=17681], [object_number=36, tag_tid=8, entry_tid=17680], [object_number=39, tag_tid=8, entry_tid=17679], [object_number=38, tag_tid=8, entry_tid=17678]]], state=connected] 2024-04-24T00:43:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=37, tag_tid=8, entry_tid=17681' 2024-04-24T00:43:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=37, tag_tid=8, entry_tid=17681' 2024-04-24T00:43:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=17681' '!=' 'object_number=38, tag_tid=8, entry_tid=17562' 2024-04-24T00:43:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=17681' = 'object_number=38, tag_tid=8, entry_tid=17562' 2024-04-24T00:43:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=37, tag_tid=8, entry_tid=17681' 2024-04-24T00:43:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=17681' 2024-04-24T00:43:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=17681' 2024-04-24T00:43:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17681 2024-04-24T00:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 17681 2024-04-24T00:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=17805], [object_number=36, tag_tid=8, entry_tid=17804], [object_number=39, tag_tid=8, entry_tid=17803], [object_number=38, tag_tid=8, entry_tid=17802]]], state=connected] 2024-04-24T00:43:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=17805], [object_number=36, tag_tid=8, entry_tid=17804], [object_number=39, tag_tid=8, entry_tid=17803], [object_number=38, tag_tid=8, entry_tid=17802]]], state=connected] 2024-04-24T00:43:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=37, tag_tid=8, entry_tid=17805' 2024-04-24T00:43:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=37, tag_tid=8, entry_tid=17805' 2024-04-24T00:43:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=17805' '!=' 'object_number=37, tag_tid=8, entry_tid=17681' 2024-04-24T00:43:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=17805' = 'object_number=37, tag_tid=8, entry_tid=17681' 2024-04-24T00:43:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=37, tag_tid=8, entry_tid=17805' 2024-04-24T00:43:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=17805' 2024-04-24T00:43:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=17805' 2024-04-24T00:43:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17805 2024-04-24T00:43:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 17805 2024-04-24T00:43:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=17919], [object_number=38, tag_tid=8, entry_tid=17918], [object_number=37, tag_tid=8, entry_tid=17917], [object_number=36, tag_tid=8, entry_tid=17916]]], state=connected] 2024-04-24T00:43:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=39, tag_tid=8, entry_tid=17919], [object_number=38, tag_tid=8, entry_tid=17918], [object_number=37, tag_tid=8, entry_tid=17917], [object_number=36, tag_tid=8, entry_tid=17916]]], state=connected] 2024-04-24T00:43:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=39, tag_tid=8, entry_tid=17919' 2024-04-24T00:43:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=39, tag_tid=8, entry_tid=17919' 2024-04-24T00:43:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=17919' '!=' 'object_number=37, tag_tid=8, entry_tid=17805' 2024-04-24T00:43:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=39, tag_tid=8, entry_tid=17919' = 'object_number=37, tag_tid=8, entry_tid=17805' 2024-04-24T00:43:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=39, tag_tid=8, entry_tid=17919' 2024-04-24T00:43:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=17919' 2024-04-24T00:43:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=39, tag_tid=8, entry_tid=17919' 2024-04-24T00:43:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17919 2024-04-24T00:43:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 17919 2024-04-24T00:43:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=18037], [object_number=36, tag_tid=8, entry_tid=18036], [object_number=39, tag_tid=8, entry_tid=18035], [object_number=38, tag_tid=8, entry_tid=18034]]], state=connected] 2024-04-24T00:43:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=18037], [object_number=36, tag_tid=8, entry_tid=18036], [object_number=39, tag_tid=8, entry_tid=18035], [object_number=38, tag_tid=8, entry_tid=18034]]], state=connected] 2024-04-24T00:43:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=37, tag_tid=8, entry_tid=18037' 2024-04-24T00:43:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=37, tag_tid=8, entry_tid=18037' 2024-04-24T00:43:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=18037' '!=' 'object_number=39, tag_tid=8, entry_tid=17919' 2024-04-24T00:43:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=18037' = 'object_number=39, tag_tid=8, entry_tid=17919' 2024-04-24T00:43:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=37, tag_tid=8, entry_tid=18037' 2024-04-24T00:43:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=18037' 2024-04-24T00:43:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=18037' 2024-04-24T00:43:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18037 2024-04-24T00:43:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 18037 2024-04-24T00:43:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=18180], [object_number=39, tag_tid=8, entry_tid=18179], [object_number=38, tag_tid=8, entry_tid=18178], [object_number=37, tag_tid=8, entry_tid=18177]]], state=connected] 2024-04-24T00:43:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=36, tag_tid=8, entry_tid=18180], [object_number=39, tag_tid=8, entry_tid=18179], [object_number=38, tag_tid=8, entry_tid=18178], [object_number=37, tag_tid=8, entry_tid=18177]]], state=connected] 2024-04-24T00:43:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=36, tag_tid=8, entry_tid=18180' 2024-04-24T00:43:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=36, tag_tid=8, entry_tid=18180' 2024-04-24T00:43:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=18180' '!=' 'object_number=37, tag_tid=8, entry_tid=18037' 2024-04-24T00:43:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=36, tag_tid=8, entry_tid=18180' = 'object_number=37, tag_tid=8, entry_tid=18037' 2024-04-24T00:43:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=36, tag_tid=8, entry_tid=18180' 2024-04-24T00:43:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=18180' 2024-04-24T00:43:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=36, tag_tid=8, entry_tid=18180' 2024-04-24T00:43:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18180 2024-04-24T00:43:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 18180 2024-04-24T00:43:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=18321], [object_number=36, tag_tid=8, entry_tid=18320], [object_number=39, tag_tid=8, entry_tid=18319], [object_number=38, tag_tid=8, entry_tid=18318]]], state=connected] 2024-04-24T00:43:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 9 2024-04-24T00:43:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=37, tag_tid=8, entry_tid=18321], [object_number=36, tag_tid=8, entry_tid=18320], [object_number=39, tag_tid=8, entry_tid=18319], [object_number=38, tag_tid=8, entry_tid=18318]]], state=connected] 2024-04-24T00:43:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=37, tag_tid=8, entry_tid=18321' 2024-04-24T00:43:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=37, tag_tid=8, entry_tid=18321' 2024-04-24T00:43:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=18321' '!=' 'object_number=36, tag_tid=8, entry_tid=18180' 2024-04-24T00:43:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=37, tag_tid=8, entry_tid=18321' = 'object_number=36, tag_tid=8, entry_tid=18180' 2024-04-24T00:43:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=37, tag_tid=8, entry_tid=18321' 2024-04-24T00:43:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=18321' 2024-04-24T00:43:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=37, tag_tid=8, entry_tid=18321' 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18321 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 18321 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=18415], [object_number=42, tag_tid=8, entry_tid=18414], [object_number=41, tag_tid=8, entry_tid=18413], [object_number=40, tag_tid=8, entry_tid=18412]]], state=connected] 2024-04-24T00:43:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=18415], [object_number=42, tag_tid=8, entry_tid=18414], [object_number=41, tag_tid=8, entry_tid=18413], [object_number=40, tag_tid=8, entry_tid=18412]]], state=connected] 2024-04-24T00:43:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=18415' 2024-04-24T00:43:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=18415' 2024-04-24T00:43:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=18415' '!=' 'object_number=37, tag_tid=8, entry_tid=18321' 2024-04-24T00:43:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=18415' = 'object_number=37, tag_tid=8, entry_tid=18321' 2024-04-24T00:43:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=18415' 2024-04-24T00:43:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=18415' 2024-04-24T00:43:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=18415' 2024-04-24T00:43:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18415 2024-04-24T00:43:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 18415 2024-04-24T00:43:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=18518], [object_number=41, tag_tid=8, entry_tid=18517], [object_number=40, tag_tid=8, entry_tid=18516], [object_number=43, tag_tid=8, entry_tid=18515]]], state=connected] 2024-04-24T00:43:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=18518], [object_number=41, tag_tid=8, entry_tid=18517], [object_number=40, tag_tid=8, entry_tid=18516], [object_number=43, tag_tid=8, entry_tid=18515]]], state=connected] 2024-04-24T00:43:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=18518' 2024-04-24T00:43:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=18518' 2024-04-24T00:43:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=18518' '!=' 'object_number=43, tag_tid=8, entry_tid=18415' 2024-04-24T00:43:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=18518' = 'object_number=43, tag_tid=8, entry_tid=18415' 2024-04-24T00:43:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=18518' 2024-04-24T00:43:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=18518' 2024-04-24T00:43:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=18518' 2024-04-24T00:43:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18518 2024-04-24T00:43:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 18518 2024-04-24T00:43:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=18624], [object_number=43, tag_tid=8, entry_tid=18623], [object_number=42, tag_tid=8, entry_tid=18622], [object_number=41, tag_tid=8, entry_tid=18621]]], state=connected] 2024-04-24T00:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=18624], [object_number=43, tag_tid=8, entry_tid=18623], [object_number=42, tag_tid=8, entry_tid=18622], [object_number=41, tag_tid=8, entry_tid=18621]]], state=connected] 2024-04-24T00:43:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=40, tag_tid=8, entry_tid=18624' 2024-04-24T00:43:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=40, tag_tid=8, entry_tid=18624' 2024-04-24T00:43:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=18624' '!=' 'object_number=42, tag_tid=8, entry_tid=18518' 2024-04-24T00:43:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=18624' = 'object_number=42, tag_tid=8, entry_tid=18518' 2024-04-24T00:43:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=40, tag_tid=8, entry_tid=18624' 2024-04-24T00:43:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=18624' 2024-04-24T00:43:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=18624' 2024-04-24T00:43:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18624 2024-04-24T00:43:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 18624 2024-04-24T00:43:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:44.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:44.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=18737], [object_number=40, tag_tid=8, entry_tid=18736], [object_number=43, tag_tid=8, entry_tid=18735], [object_number=42, tag_tid=8, entry_tid=18734]]], state=connected] 2024-04-24T00:43:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=18737], [object_number=40, tag_tid=8, entry_tid=18736], [object_number=43, tag_tid=8, entry_tid=18735], [object_number=42, tag_tid=8, entry_tid=18734]]], state=connected] 2024-04-24T00:43:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=41, tag_tid=8, entry_tid=18737' 2024-04-24T00:43:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=8, entry_tid=18737' 2024-04-24T00:43:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=18737' '!=' 'object_number=40, tag_tid=8, entry_tid=18624' 2024-04-24T00:43:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=18737' = 'object_number=40, tag_tid=8, entry_tid=18624' 2024-04-24T00:43:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=41, tag_tid=8, entry_tid=18737' 2024-04-24T00:43:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=18737' 2024-04-24T00:43:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=18737' 2024-04-24T00:43:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18737 2024-04-24T00:43:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 18737 2024-04-24T00:43:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=18851], [object_number=42, tag_tid=8, entry_tid=18850], [object_number=41, tag_tid=8, entry_tid=18849], [object_number=40, tag_tid=8, entry_tid=18848]]], state=connected] 2024-04-24T00:43:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=18851], [object_number=42, tag_tid=8, entry_tid=18850], [object_number=41, tag_tid=8, entry_tid=18849], [object_number=40, tag_tid=8, entry_tid=18848]]], state=connected] 2024-04-24T00:43:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=18851' 2024-04-24T00:43:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=18851' 2024-04-24T00:43:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=18851' '!=' 'object_number=41, tag_tid=8, entry_tid=18737' 2024-04-24T00:43:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=18851' = 'object_number=41, tag_tid=8, entry_tid=18737' 2024-04-24T00:43:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=18851' 2024-04-24T00:43:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=18851' 2024-04-24T00:43:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=18851' 2024-04-24T00:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18851 2024-04-24T00:43:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 18851 2024-04-24T00:43:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:45.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=18974], [object_number=41, tag_tid=8, entry_tid=18973], [object_number=40, tag_tid=8, entry_tid=18972], [object_number=43, tag_tid=8, entry_tid=18971]]], state=connected] 2024-04-24T00:43:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=18974], [object_number=41, tag_tid=8, entry_tid=18973], [object_number=40, tag_tid=8, entry_tid=18972], [object_number=43, tag_tid=8, entry_tid=18971]]], state=connected] 2024-04-24T00:43:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=18974' 2024-04-24T00:43:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=18974' 2024-04-24T00:43:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=18974' '!=' 'object_number=43, tag_tid=8, entry_tid=18851' 2024-04-24T00:43:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=18974' = 'object_number=43, tag_tid=8, entry_tid=18851' 2024-04-24T00:43:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=18974' 2024-04-24T00:43:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=18974' 2024-04-24T00:43:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=18974' 2024-04-24T00:43:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18974 2024-04-24T00:43:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 18974 2024-04-24T00:43:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=19099], [object_number=42, tag_tid=8, entry_tid=19098], [object_number=41, tag_tid=8, entry_tid=19097], [object_number=40, tag_tid=8, entry_tid=19096]]], state=connected] 2024-04-24T00:43:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=19099], [object_number=42, tag_tid=8, entry_tid=19098], [object_number=41, tag_tid=8, entry_tid=19097], [object_number=40, tag_tid=8, entry_tid=19096]]], state=connected] 2024-04-24T00:43:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=19099' 2024-04-24T00:43:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=19099' 2024-04-24T00:43:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=19099' '!=' 'object_number=42, tag_tid=8, entry_tid=18974' 2024-04-24T00:43:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=19099' = 'object_number=42, tag_tid=8, entry_tid=18974' 2024-04-24T00:43:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=19099' 2024-04-24T00:43:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=19099' 2024-04-24T00:43:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=19099' 2024-04-24T00:43:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19099 2024-04-24T00:43:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 19099 2024-04-24T00:43:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=19214], [object_number=41, tag_tid=8, entry_tid=19213], [object_number=40, tag_tid=8, entry_tid=19212], [object_number=43, tag_tid=8, entry_tid=19211]]], state=connected] 2024-04-24T00:43:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=19214], [object_number=41, tag_tid=8, entry_tid=19213], [object_number=40, tag_tid=8, entry_tid=19212], [object_number=43, tag_tid=8, entry_tid=19211]]], state=connected] 2024-04-24T00:43:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=19214' 2024-04-24T00:43:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=19214' 2024-04-24T00:43:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=19214' '!=' 'object_number=43, tag_tid=8, entry_tid=19099' 2024-04-24T00:43:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=19214' = 'object_number=43, tag_tid=8, entry_tid=19099' 2024-04-24T00:43:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=19214' 2024-04-24T00:43:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=19214' 2024-04-24T00:43:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=19214' 2024-04-24T00:43:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19214 2024-04-24T00:43:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 19214 2024-04-24T00:43:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=19344], [object_number=43, tag_tid=8, entry_tid=19343], [object_number=42, tag_tid=8, entry_tid=19342], [object_number=41, tag_tid=8, entry_tid=19341]]], state=connected] 2024-04-24T00:43:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=19344], [object_number=43, tag_tid=8, entry_tid=19343], [object_number=42, tag_tid=8, entry_tid=19342], [object_number=41, tag_tid=8, entry_tid=19341]]], state=connected] 2024-04-24T00:43:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=40, tag_tid=8, entry_tid=19344' 2024-04-24T00:43:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=40, tag_tid=8, entry_tid=19344' 2024-04-24T00:43:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=19344' '!=' 'object_number=42, tag_tid=8, entry_tid=19214' 2024-04-24T00:43:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=19344' = 'object_number=42, tag_tid=8, entry_tid=19214' 2024-04-24T00:43:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=40, tag_tid=8, entry_tid=19344' 2024-04-24T00:43:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=19344' 2024-04-24T00:43:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=19344' 2024-04-24T00:43:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19344 2024-04-24T00:43:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 19344 2024-04-24T00:43:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=19470], [object_number=41, tag_tid=8, entry_tid=19469], [object_number=40, tag_tid=8, entry_tid=19468], [object_number=43, tag_tid=8, entry_tid=19467]]], state=connected] 2024-04-24T00:43:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=19470], [object_number=41, tag_tid=8, entry_tid=19469], [object_number=40, tag_tid=8, entry_tid=19468], [object_number=43, tag_tid=8, entry_tid=19467]]], state=connected] 2024-04-24T00:43:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=19470' 2024-04-24T00:43:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=19470' 2024-04-24T00:43:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=19470' '!=' 'object_number=40, tag_tid=8, entry_tid=19344' 2024-04-24T00:43:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=19470' = 'object_number=40, tag_tid=8, entry_tid=19344' 2024-04-24T00:43:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=19470' 2024-04-24T00:43:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=19470' 2024-04-24T00:43:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=19470' 2024-04-24T00:43:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19470 2024-04-24T00:43:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 19470 2024-04-24T00:43:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=19597], [object_number=40, tag_tid=8, entry_tid=19596], [object_number=43, tag_tid=8, entry_tid=19595], [object_number=42, tag_tid=8, entry_tid=19594]]], state=connected] 2024-04-24T00:43:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=19597], [object_number=40, tag_tid=8, entry_tid=19596], [object_number=43, tag_tid=8, entry_tid=19595], [object_number=42, tag_tid=8, entry_tid=19594]]], state=connected] 2024-04-24T00:43:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=41, tag_tid=8, entry_tid=19597' 2024-04-24T00:43:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=8, entry_tid=19597' 2024-04-24T00:43:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=19597' '!=' 'object_number=42, tag_tid=8, entry_tid=19470' 2024-04-24T00:43:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=19597' = 'object_number=42, tag_tid=8, entry_tid=19470' 2024-04-24T00:43:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=41, tag_tid=8, entry_tid=19597' 2024-04-24T00:43:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=19597' 2024-04-24T00:43:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=19597' 2024-04-24T00:43:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19597 2024-04-24T00:43:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 19597 2024-04-24T00:43:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=19716], [object_number=43, tag_tid=8, entry_tid=19715], [object_number=42, tag_tid=8, entry_tid=19714], [object_number=41, tag_tid=8, entry_tid=19713]]], state=connected] 2024-04-24T00:43:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=19716], [object_number=43, tag_tid=8, entry_tid=19715], [object_number=42, tag_tid=8, entry_tid=19714], [object_number=41, tag_tid=8, entry_tid=19713]]], state=connected] 2024-04-24T00:43:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=40, tag_tid=8, entry_tid=19716' 2024-04-24T00:43:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=40, tag_tid=8, entry_tid=19716' 2024-04-24T00:43:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=19716' '!=' 'object_number=41, tag_tid=8, entry_tid=19597' 2024-04-24T00:43:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=19716' = 'object_number=41, tag_tid=8, entry_tid=19597' 2024-04-24T00:43:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=40, tag_tid=8, entry_tid=19716' 2024-04-24T00:43:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=19716' 2024-04-24T00:43:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=19716' 2024-04-24T00:43:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19716 2024-04-24T00:43:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 19716 2024-04-24T00:43:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=19846], [object_number=41, tag_tid=8, entry_tid=19845], [object_number=40, tag_tid=8, entry_tid=19844], [object_number=43, tag_tid=8, entry_tid=19843]]], state=connected] 2024-04-24T00:43:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=19846], [object_number=41, tag_tid=8, entry_tid=19845], [object_number=40, tag_tid=8, entry_tid=19844], [object_number=43, tag_tid=8, entry_tid=19843]]], state=connected] 2024-04-24T00:43:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=19846' 2024-04-24T00:43:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=19846' 2024-04-24T00:43:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=19846' '!=' 'object_number=40, tag_tid=8, entry_tid=19716' 2024-04-24T00:43:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=19846' = 'object_number=40, tag_tid=8, entry_tid=19716' 2024-04-24T00:43:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=19846' 2024-04-24T00:43:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=19846' 2024-04-24T00:43:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=19846' 2024-04-24T00:43:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19846 2024-04-24T00:43:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 19846 2024-04-24T00:43:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=19966], [object_number=41, tag_tid=8, entry_tid=19965], [object_number=40, tag_tid=8, entry_tid=19964], [object_number=43, tag_tid=8, entry_tid=19963]]], state=connected] 2024-04-24T00:43:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=19966], [object_number=41, tag_tid=8, entry_tid=19965], [object_number=40, tag_tid=8, entry_tid=19964], [object_number=43, tag_tid=8, entry_tid=19963]]], state=connected] 2024-04-24T00:43:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=19966' 2024-04-24T00:43:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=19966' 2024-04-24T00:43:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=19966' '!=' 'object_number=42, tag_tid=8, entry_tid=19846' 2024-04-24T00:43:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=19966' = 'object_number=42, tag_tid=8, entry_tid=19846' 2024-04-24T00:43:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=19966' 2024-04-24T00:43:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=19966' 2024-04-24T00:43:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=19966' 2024-04-24T00:43:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19966 2024-04-24T00:43:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 19966 2024-04-24T00:43:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=20086], [object_number=41, tag_tid=8, entry_tid=20085], [object_number=40, tag_tid=8, entry_tid=20084], [object_number=43, tag_tid=8, entry_tid=20083]]], state=connected] 2024-04-24T00:43:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=20086], [object_number=41, tag_tid=8, entry_tid=20085], [object_number=40, tag_tid=8, entry_tid=20084], [object_number=43, tag_tid=8, entry_tid=20083]]], state=connected] 2024-04-24T00:43:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=20086' 2024-04-24T00:43:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=20086' 2024-04-24T00:43:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=20086' '!=' 'object_number=42, tag_tid=8, entry_tid=19966' 2024-04-24T00:43:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=20086' = 'object_number=42, tag_tid=8, entry_tid=19966' 2024-04-24T00:43:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=20086' 2024-04-24T00:43:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=20086' 2024-04-24T00:43:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=20086' 2024-04-24T00:43:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20086 2024-04-24T00:43:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 20086 2024-04-24T00:43:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=20201], [object_number=40, tag_tid=8, entry_tid=20200], [object_number=43, tag_tid=8, entry_tid=20199], [object_number=42, tag_tid=8, entry_tid=20198]]], state=connected] 2024-04-24T00:43:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=20201], [object_number=40, tag_tid=8, entry_tid=20200], [object_number=43, tag_tid=8, entry_tid=20199], [object_number=42, tag_tid=8, entry_tid=20198]]], state=connected] 2024-04-24T00:43:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=41, tag_tid=8, entry_tid=20201' 2024-04-24T00:43:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=8, entry_tid=20201' 2024-04-24T00:43:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=20201' '!=' 'object_number=42, tag_tid=8, entry_tid=20086' 2024-04-24T00:43:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=20201' = 'object_number=42, tag_tid=8, entry_tid=20086' 2024-04-24T00:43:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=41, tag_tid=8, entry_tid=20201' 2024-04-24T00:43:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=20201' 2024-04-24T00:43:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=20201' 2024-04-24T00:43:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20201 2024-04-24T00:43:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 20201 2024-04-24T00:43:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=20339], [object_number=42, tag_tid=8, entry_tid=20338], [object_number=41, tag_tid=8, entry_tid=20337], [object_number=40, tag_tid=8, entry_tid=20336]]], state=connected] 2024-04-24T00:43:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=20339], [object_number=42, tag_tid=8, entry_tid=20338], [object_number=41, tag_tid=8, entry_tid=20337], [object_number=40, tag_tid=8, entry_tid=20336]]], state=connected] 2024-04-24T00:43:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=20339' 2024-04-24T00:43:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=20339' 2024-04-24T00:43:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=20339' '!=' 'object_number=41, tag_tid=8, entry_tid=20201' 2024-04-24T00:43:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=20339' = 'object_number=41, tag_tid=8, entry_tid=20201' 2024-04-24T00:43:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=20339' 2024-04-24T00:43:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=20339' 2024-04-24T00:43:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=20339' 2024-04-24T00:43:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20339 2024-04-24T00:43:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 20339 2024-04-24T00:43:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=20470], [object_number=41, tag_tid=8, entry_tid=20469], [object_number=40, tag_tid=8, entry_tid=20468], [object_number=43, tag_tid=8, entry_tid=20467]]], state=connected] 2024-04-24T00:43:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=20470], [object_number=41, tag_tid=8, entry_tid=20469], [object_number=40, tag_tid=8, entry_tid=20468], [object_number=43, tag_tid=8, entry_tid=20467]]], state=connected] 2024-04-24T00:43:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=20470' 2024-04-24T00:43:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=20470' 2024-04-24T00:43:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=20470' '!=' 'object_number=43, tag_tid=8, entry_tid=20339' 2024-04-24T00:43:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=20470' = 'object_number=43, tag_tid=8, entry_tid=20339' 2024-04-24T00:43:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=20470' 2024-04-24T00:43:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=20470' 2024-04-24T00:43:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=20470' 2024-04-24T00:43:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20470 2024-04-24T00:43:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 20470 2024-04-24T00:43:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=20684], [object_number=43, tag_tid=8, entry_tid=20683], [object_number=42, tag_tid=8, entry_tid=20682], [object_number=41, tag_tid=8, entry_tid=20681]]], state=connected] 2024-04-24T00:43:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=20684], [object_number=43, tag_tid=8, entry_tid=20683], [object_number=42, tag_tid=8, entry_tid=20682], [object_number=41, tag_tid=8, entry_tid=20681]]], state=connected] 2024-04-24T00:43:54.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=40, tag_tid=8, entry_tid=20684' 2024-04-24T00:43:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=40, tag_tid=8, entry_tid=20684' 2024-04-24T00:43:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=20684' '!=' 'object_number=42, tag_tid=8, entry_tid=20470' 2024-04-24T00:43:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=20684' = 'object_number=42, tag_tid=8, entry_tid=20470' 2024-04-24T00:43:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=40, tag_tid=8, entry_tid=20684' 2024-04-24T00:43:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=20684' 2024-04-24T00:43:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=20684' 2024-04-24T00:43:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20684 2024-04-24T00:43:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 20684 2024-04-24T00:43:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=20801], [object_number=40, tag_tid=8, entry_tid=20800], [object_number=43, tag_tid=8, entry_tid=20799], [object_number=42, tag_tid=8, entry_tid=20798]]], state=connected] 2024-04-24T00:43:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=20801], [object_number=40, tag_tid=8, entry_tid=20800], [object_number=43, tag_tid=8, entry_tid=20799], [object_number=42, tag_tid=8, entry_tid=20798]]], state=connected] 2024-04-24T00:43:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=41, tag_tid=8, entry_tid=20801' 2024-04-24T00:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=8, entry_tid=20801' 2024-04-24T00:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=20801' '!=' 'object_number=40, tag_tid=8, entry_tid=20684' 2024-04-24T00:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=20801' = 'object_number=40, tag_tid=8, entry_tid=20684' 2024-04-24T00:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=41, tag_tid=8, entry_tid=20801' 2024-04-24T00:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=20801' 2024-04-24T00:43:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=20801' 2024-04-24T00:43:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20801 2024-04-24T00:43:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 20801 2024-04-24T00:43:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:55.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=20914], [object_number=41, tag_tid=8, entry_tid=20913], [object_number=40, tag_tid=8, entry_tid=20912], [object_number=43, tag_tid=8, entry_tid=20911]]], state=connected] 2024-04-24T00:43:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=20914], [object_number=41, tag_tid=8, entry_tid=20913], [object_number=40, tag_tid=8, entry_tid=20912], [object_number=43, tag_tid=8, entry_tid=20911]]], state=connected] 2024-04-24T00:43:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=20914' 2024-04-24T00:43:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=20914' 2024-04-24T00:43:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=20914' '!=' 'object_number=41, tag_tid=8, entry_tid=20801' 2024-04-24T00:43:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=20914' = 'object_number=41, tag_tid=8, entry_tid=20801' 2024-04-24T00:43:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=20914' 2024-04-24T00:43:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=20914' 2024-04-24T00:43:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=20914' 2024-04-24T00:43:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20914 2024-04-24T00:43:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 20914 2024-04-24T00:43:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=21027], [object_number=42, tag_tid=8, entry_tid=21026], [object_number=41, tag_tid=8, entry_tid=21025], [object_number=40, tag_tid=8, entry_tid=21024]]], state=connected] 2024-04-24T00:43:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=21027], [object_number=42, tag_tid=8, entry_tid=21026], [object_number=41, tag_tid=8, entry_tid=21025], [object_number=40, tag_tid=8, entry_tid=21024]]], state=connected] 2024-04-24T00:43:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=21027' 2024-04-24T00:43:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=21027' 2024-04-24T00:43:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=21027' '!=' 'object_number=42, tag_tid=8, entry_tid=20914' 2024-04-24T00:43:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=21027' = 'object_number=42, tag_tid=8, entry_tid=20914' 2024-04-24T00:43:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=21027' 2024-04-24T00:43:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=21027' 2024-04-24T00:43:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=21027' 2024-04-24T00:43:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21027 2024-04-24T00:43:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 21027 2024-04-24T00:43:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=21138], [object_number=41, tag_tid=8, entry_tid=21137], [object_number=40, tag_tid=8, entry_tid=21136], [object_number=43, tag_tid=8, entry_tid=21135]]], state=connected] 2024-04-24T00:43:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=21138], [object_number=41, tag_tid=8, entry_tid=21137], [object_number=40, tag_tid=8, entry_tid=21136], [object_number=43, tag_tid=8, entry_tid=21135]]], state=connected] 2024-04-24T00:43:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=21138' 2024-04-24T00:43:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=21138' 2024-04-24T00:43:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=21138' '!=' 'object_number=43, tag_tid=8, entry_tid=21027' 2024-04-24T00:43:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=21138' = 'object_number=43, tag_tid=8, entry_tid=21027' 2024-04-24T00:43:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=21138' 2024-04-24T00:43:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=21138' 2024-04-24T00:43:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=21138' 2024-04-24T00:43:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21138 2024-04-24T00:43:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 21138 2024-04-24T00:43:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=21297], [object_number=40, tag_tid=8, entry_tid=21296], [object_number=43, tag_tid=8, entry_tid=21295], [object_number=42, tag_tid=8, entry_tid=21294]]], state=connected] 2024-04-24T00:43:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=21297], [object_number=40, tag_tid=8, entry_tid=21296], [object_number=43, tag_tid=8, entry_tid=21295], [object_number=42, tag_tid=8, entry_tid=21294]]], state=connected] 2024-04-24T00:43:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=41, tag_tid=8, entry_tid=21297' 2024-04-24T00:43:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=8, entry_tid=21297' 2024-04-24T00:43:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=21297' '!=' 'object_number=42, tag_tid=8, entry_tid=21138' 2024-04-24T00:43:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=21297' = 'object_number=42, tag_tid=8, entry_tid=21138' 2024-04-24T00:43:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=41, tag_tid=8, entry_tid=21297' 2024-04-24T00:43:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=21297' 2024-04-24T00:43:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=21297' 2024-04-24T00:43:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21297 2024-04-24T00:43:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 21297 2024-04-24T00:43:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=21422], [object_number=41, tag_tid=8, entry_tid=21421], [object_number=40, tag_tid=8, entry_tid=21420], [object_number=43, tag_tid=8, entry_tid=21419]]], state=connected] 2024-04-24T00:43:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=21422], [object_number=41, tag_tid=8, entry_tid=21421], [object_number=40, tag_tid=8, entry_tid=21420], [object_number=43, tag_tid=8, entry_tid=21419]]], state=connected] 2024-04-24T00:43:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=21422' 2024-04-24T00:43:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=21422' 2024-04-24T00:43:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=21422' '!=' 'object_number=41, tag_tid=8, entry_tid=21297' 2024-04-24T00:43:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=21422' = 'object_number=41, tag_tid=8, entry_tid=21297' 2024-04-24T00:43:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=21422' 2024-04-24T00:43:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=21422' 2024-04-24T00:43:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=21422' 2024-04-24T00:43:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21422 2024-04-24T00:43:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 21422 2024-04-24T00:43:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=21538], [object_number=41, tag_tid=8, entry_tid=21537], [object_number=40, tag_tid=8, entry_tid=21536], [object_number=43, tag_tid=8, entry_tid=21535]]], state=connected] 2024-04-24T00:43:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=21538], [object_number=41, tag_tid=8, entry_tid=21537], [object_number=40, tag_tid=8, entry_tid=21536], [object_number=43, tag_tid=8, entry_tid=21535]]], state=connected] 2024-04-24T00:43:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=21538' 2024-04-24T00:43:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=21538' 2024-04-24T00:43:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=21538' '!=' 'object_number=42, tag_tid=8, entry_tid=21422' 2024-04-24T00:43:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=21538' = 'object_number=42, tag_tid=8, entry_tid=21422' 2024-04-24T00:43:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=21538' 2024-04-24T00:43:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=21538' 2024-04-24T00:43:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=21538' 2024-04-24T00:43:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21538 2024-04-24T00:43:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 21538 2024-04-24T00:43:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:43:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:43:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:43:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:43:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:43:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:43:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:43:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:43:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:43:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:43:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:43:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:43:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:43:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:43:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:43:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:43:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:43:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:43:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:43:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:43:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=21663], [object_number=42, tag_tid=8, entry_tid=21662], [object_number=41, tag_tid=8, entry_tid=21661], [object_number=40, tag_tid=8, entry_tid=21660]]], state=connected] 2024-04-24T00:43:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:43:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:43:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:43:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:43:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:43:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:43:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:43:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:43:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:43:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:43:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:43:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:43:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:43:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:43:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=21663], [object_number=42, tag_tid=8, entry_tid=21662], [object_number=41, tag_tid=8, entry_tid=21661], [object_number=40, tag_tid=8, entry_tid=21660]]], state=connected] 2024-04-24T00:43:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:43:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=21663' 2024-04-24T00:43:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=21663' 2024-04-24T00:43:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=21663' '!=' 'object_number=42, tag_tid=8, entry_tid=21538' 2024-04-24T00:43:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:43:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=21663' = 'object_number=42, tag_tid=8, entry_tid=21538' 2024-04-24T00:43:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=21663' 2024-04-24T00:43:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:43:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=21663' 2024-04-24T00:43:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:43:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:43:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:43:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:43:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=21663' 2024-04-24T00:43:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:43:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:43:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21663 2024-04-24T00:43:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 21663 2024-04-24T00:43:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:43:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:43:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:43:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:43:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:43:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:43:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:43:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:43:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:43:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:43:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:43:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:43:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:43:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:43:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:43:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:43:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:43:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:43:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:43:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=21789], [object_number=40, tag_tid=8, entry_tid=21788], [object_number=43, tag_tid=8, entry_tid=21787], [object_number=42, tag_tid=8, entry_tid=21786]]], state=connected] 2024-04-24T00:44:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=21789], [object_number=40, tag_tid=8, entry_tid=21788], [object_number=43, tag_tid=8, entry_tid=21787], [object_number=42, tag_tid=8, entry_tid=21786]]], state=connected] 2024-04-24T00:44:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=41, tag_tid=8, entry_tid=21789' 2024-04-24T00:44:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=8, entry_tid=21789' 2024-04-24T00:44:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=21789' '!=' 'object_number=43, tag_tid=8, entry_tid=21663' 2024-04-24T00:44:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=21789' = 'object_number=43, tag_tid=8, entry_tid=21663' 2024-04-24T00:44:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=41, tag_tid=8, entry_tid=21789' 2024-04-24T00:44:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=21789' 2024-04-24T00:44:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=21789' 2024-04-24T00:44:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21789 2024-04-24T00:44:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 21789 2024-04-24T00:44:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=21904], [object_number=43, tag_tid=8, entry_tid=21903], [object_number=42, tag_tid=8, entry_tid=21902], [object_number=41, tag_tid=8, entry_tid=21901]]], state=connected] 2024-04-24T00:44:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=21904], [object_number=43, tag_tid=8, entry_tid=21903], [object_number=42, tag_tid=8, entry_tid=21902], [object_number=41, tag_tid=8, entry_tid=21901]]], state=connected] 2024-04-24T00:44:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=40, tag_tid=8, entry_tid=21904' 2024-04-24T00:44:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=40, tag_tid=8, entry_tid=21904' 2024-04-24T00:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=21904' '!=' 'object_number=41, tag_tid=8, entry_tid=21789' 2024-04-24T00:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=21904' = 'object_number=41, tag_tid=8, entry_tid=21789' 2024-04-24T00:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=40, tag_tid=8, entry_tid=21904' 2024-04-24T00:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=21904' 2024-04-24T00:44:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=21904' 2024-04-24T00:44:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21904 2024-04-24T00:44:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 21904 2024-04-24T00:44:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=22011], [object_number=42, tag_tid=8, entry_tid=22010], [object_number=41, tag_tid=8, entry_tid=22009], [object_number=40, tag_tid=8, entry_tid=22008]]], state=connected] 2024-04-24T00:44:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=22011], [object_number=42, tag_tid=8, entry_tid=22010], [object_number=41, tag_tid=8, entry_tid=22009], [object_number=40, tag_tid=8, entry_tid=22008]]], state=connected] 2024-04-24T00:44:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=22011' 2024-04-24T00:44:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=22011' 2024-04-24T00:44:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=22011' '!=' 'object_number=40, tag_tid=8, entry_tid=21904' 2024-04-24T00:44:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=22011' = 'object_number=40, tag_tid=8, entry_tid=21904' 2024-04-24T00:44:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=22011' 2024-04-24T00:44:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=22011' 2024-04-24T00:44:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=22011' 2024-04-24T00:44:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22011 2024-04-24T00:44:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 22011 2024-04-24T00:44:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:02.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=22117], [object_number=40, tag_tid=8, entry_tid=22116], [object_number=43, tag_tid=8, entry_tid=22115], [object_number=42, tag_tid=8, entry_tid=22114]]], state=connected] 2024-04-24T00:44:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=22117], [object_number=40, tag_tid=8, entry_tid=22116], [object_number=43, tag_tid=8, entry_tid=22115], [object_number=42, tag_tid=8, entry_tid=22114]]], state=connected] 2024-04-24T00:44:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=41, tag_tid=8, entry_tid=22117' 2024-04-24T00:44:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=8, entry_tid=22117' 2024-04-24T00:44:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=22117' '!=' 'object_number=43, tag_tid=8, entry_tid=22011' 2024-04-24T00:44:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=22117' = 'object_number=43, tag_tid=8, entry_tid=22011' 2024-04-24T00:44:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=41, tag_tid=8, entry_tid=22117' 2024-04-24T00:44:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=22117' 2024-04-24T00:44:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=22117' 2024-04-24T00:44:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22117 2024-04-24T00:44:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 22117 2024-04-24T00:44:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:02.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=22233], [object_number=40, tag_tid=8, entry_tid=22232], [object_number=43, tag_tid=8, entry_tid=22231], [object_number=42, tag_tid=8, entry_tid=22230]]], state=connected] 2024-04-24T00:44:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=22233], [object_number=40, tag_tid=8, entry_tid=22232], [object_number=43, tag_tid=8, entry_tid=22231], [object_number=42, tag_tid=8, entry_tid=22230]]], state=connected] 2024-04-24T00:44:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=41, tag_tid=8, entry_tid=22233' 2024-04-24T00:44:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=8, entry_tid=22233' 2024-04-24T00:44:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=22233' '!=' 'object_number=41, tag_tid=8, entry_tid=22117' 2024-04-24T00:44:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=22233' = 'object_number=41, tag_tid=8, entry_tid=22117' 2024-04-24T00:44:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=41, tag_tid=8, entry_tid=22233' 2024-04-24T00:44:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=22233' 2024-04-24T00:44:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=22233' 2024-04-24T00:44:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22233 2024-04-24T00:44:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 22233 2024-04-24T00:44:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=22339], [object_number=42, tag_tid=8, entry_tid=22338], [object_number=41, tag_tid=8, entry_tid=22337], [object_number=40, tag_tid=8, entry_tid=22336]]], state=connected] 2024-04-24T00:44:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=22339], [object_number=42, tag_tid=8, entry_tid=22338], [object_number=41, tag_tid=8, entry_tid=22337], [object_number=40, tag_tid=8, entry_tid=22336]]], state=connected] 2024-04-24T00:44:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=22339' 2024-04-24T00:44:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=22339' 2024-04-24T00:44:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=22339' '!=' 'object_number=41, tag_tid=8, entry_tid=22233' 2024-04-24T00:44:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=22339' = 'object_number=41, tag_tid=8, entry_tid=22233' 2024-04-24T00:44:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=22339' 2024-04-24T00:44:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=22339' 2024-04-24T00:44:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=22339' 2024-04-24T00:44:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22339 2024-04-24T00:44:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 22339 2024-04-24T00:44:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=22450], [object_number=41, tag_tid=8, entry_tid=22449], [object_number=40, tag_tid=8, entry_tid=22448], [object_number=43, tag_tid=8, entry_tid=22447]]], state=connected] 2024-04-24T00:44:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=22450], [object_number=41, tag_tid=8, entry_tid=22449], [object_number=40, tag_tid=8, entry_tid=22448], [object_number=43, tag_tid=8, entry_tid=22447]]], state=connected] 2024-04-24T00:44:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=22450' 2024-04-24T00:44:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=22450' 2024-04-24T00:44:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=22450' '!=' 'object_number=43, tag_tid=8, entry_tid=22339' 2024-04-24T00:44:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=22450' = 'object_number=43, tag_tid=8, entry_tid=22339' 2024-04-24T00:44:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=22450' 2024-04-24T00:44:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=22450' 2024-04-24T00:44:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=22450' 2024-04-24T00:44:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22450 2024-04-24T00:44:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 22450 2024-04-24T00:44:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=22565], [object_number=40, tag_tid=8, entry_tid=22564], [object_number=43, tag_tid=8, entry_tid=22563], [object_number=42, tag_tid=8, entry_tid=22562]]], state=connected] 2024-04-24T00:44:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 10 2024-04-24T00:44:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=22565], [object_number=40, tag_tid=8, entry_tid=22564], [object_number=43, tag_tid=8, entry_tid=22563], [object_number=42, tag_tid=8, entry_tid=22562]]], state=connected] 2024-04-24T00:44:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=22565' 2024-04-24T00:44:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=22565' 2024-04-24T00:44:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=22565' '!=' 'object_number=42, tag_tid=8, entry_tid=22450' 2024-04-24T00:44:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=22565' = 'object_number=42, tag_tid=8, entry_tid=22450' 2024-04-24T00:44:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=22565' 2024-04-24T00:44:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=22565' 2024-04-24T00:44:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=22565' 2024-04-24T00:44:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22565 2024-04-24T00:44:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 22565 2024-04-24T00:44:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=22673], [object_number=44, tag_tid=8, entry_tid=22672], [object_number=47, tag_tid=8, entry_tid=22671], [object_number=46, tag_tid=8, entry_tid=22670]]], state=connected] 2024-04-24T00:44:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=22673], [object_number=44, tag_tid=8, entry_tid=22672], [object_number=47, tag_tid=8, entry_tid=22671], [object_number=46, tag_tid=8, entry_tid=22670]]], state=connected] 2024-04-24T00:44:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=22673' 2024-04-24T00:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=22673' 2024-04-24T00:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=22673' '!=' 'object_number=45, tag_tid=8, entry_tid=22565' 2024-04-24T00:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=22673' = 'object_number=45, tag_tid=8, entry_tid=22565' 2024-04-24T00:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=22673' 2024-04-24T00:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=22673' 2024-04-24T00:44:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=22673' 2024-04-24T00:44:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22673 2024-04-24T00:44:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 22673 2024-04-24T00:44:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=22814], [object_number=45, tag_tid=8, entry_tid=22813], [object_number=44, tag_tid=8, entry_tid=22812], [object_number=47, tag_tid=8, entry_tid=22811]]], state=connected] 2024-04-24T00:44:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=22814], [object_number=45, tag_tid=8, entry_tid=22813], [object_number=44, tag_tid=8, entry_tid=22812], [object_number=47, tag_tid=8, entry_tid=22811]]], state=connected] 2024-04-24T00:44:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=22814' 2024-04-24T00:44:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=22814' 2024-04-24T00:44:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=22814' '!=' 'object_number=45, tag_tid=8, entry_tid=22673' 2024-04-24T00:44:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=22814' = 'object_number=45, tag_tid=8, entry_tid=22673' 2024-04-24T00:44:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=22814' 2024-04-24T00:44:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=22814' 2024-04-24T00:44:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=22814' 2024-04-24T00:44:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22814 2024-04-24T00:44:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 22814 2024-04-24T00:44:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=22950], [object_number=45, tag_tid=8, entry_tid=22949], [object_number=44, tag_tid=8, entry_tid=22948], [object_number=47, tag_tid=8, entry_tid=22947]]], state=connected] 2024-04-24T00:44:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=22950], [object_number=45, tag_tid=8, entry_tid=22949], [object_number=44, tag_tid=8, entry_tid=22948], [object_number=47, tag_tid=8, entry_tid=22947]]], state=connected] 2024-04-24T00:44:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=22950' 2024-04-24T00:44:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=22950' 2024-04-24T00:44:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=22950' '!=' 'object_number=46, tag_tid=8, entry_tid=22814' 2024-04-24T00:44:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=22950' = 'object_number=46, tag_tid=8, entry_tid=22814' 2024-04-24T00:44:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=22950' 2024-04-24T00:44:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=22950' 2024-04-24T00:44:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=22950' 2024-04-24T00:44:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22950 2024-04-24T00:44:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 22950 2024-04-24T00:44:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23067], [object_number=46, tag_tid=8, entry_tid=23066], [object_number=45, tag_tid=8, entry_tid=23065], [object_number=44, tag_tid=8, entry_tid=23064]]], state=connected] 2024-04-24T00:44:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23067], [object_number=46, tag_tid=8, entry_tid=23066], [object_number=45, tag_tid=8, entry_tid=23065], [object_number=44, tag_tid=8, entry_tid=23064]]], state=connected] 2024-04-24T00:44:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=23067' 2024-04-24T00:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=23067' 2024-04-24T00:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23067' '!=' 'object_number=46, tag_tid=8, entry_tid=22950' 2024-04-24T00:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23067' = 'object_number=46, tag_tid=8, entry_tid=22950' 2024-04-24T00:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=23067' 2024-04-24T00:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23067' 2024-04-24T00:44:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23067' 2024-04-24T00:44:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=23067 2024-04-24T00:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 23067 2024-04-24T00:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23183], [object_number=46, tag_tid=8, entry_tid=23182], [object_number=45, tag_tid=8, entry_tid=23181], [object_number=44, tag_tid=8, entry_tid=23180]]], state=connected] 2024-04-24T00:44:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23183], [object_number=46, tag_tid=8, entry_tid=23182], [object_number=45, tag_tid=8, entry_tid=23181], [object_number=44, tag_tid=8, entry_tid=23180]]], state=connected] 2024-04-24T00:44:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=23183' 2024-04-24T00:44:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=23183' 2024-04-24T00:44:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23183' '!=' 'object_number=47, tag_tid=8, entry_tid=23067' 2024-04-24T00:44:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23183' = 'object_number=47, tag_tid=8, entry_tid=23067' 2024-04-24T00:44:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=23183' 2024-04-24T00:44:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23183' 2024-04-24T00:44:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23183' 2024-04-24T00:44:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=23183 2024-04-24T00:44:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 23183 2024-04-24T00:44:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23295], [object_number=46, tag_tid=8, entry_tid=23294], [object_number=45, tag_tid=8, entry_tid=23293], [object_number=44, tag_tid=8, entry_tid=23292]]], state=connected] 2024-04-24T00:44:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23295], [object_number=46, tag_tid=8, entry_tid=23294], [object_number=45, tag_tid=8, entry_tid=23293], [object_number=44, tag_tid=8, entry_tid=23292]]], state=connected] 2024-04-24T00:44:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=23295' 2024-04-24T00:44:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=23295' 2024-04-24T00:44:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23295' '!=' 'object_number=47, tag_tid=8, entry_tid=23183' 2024-04-24T00:44:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23295' = 'object_number=47, tag_tid=8, entry_tid=23183' 2024-04-24T00:44:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=23295' 2024-04-24T00:44:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23295' 2024-04-24T00:44:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23295' 2024-04-24T00:44:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=23295 2024-04-24T00:44:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 23295 2024-04-24T00:44:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=23413], [object_number=44, tag_tid=8, entry_tid=23412], [object_number=47, tag_tid=8, entry_tid=23411], [object_number=46, tag_tid=8, entry_tid=23410]]], state=connected] 2024-04-24T00:44:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=23413], [object_number=44, tag_tid=8, entry_tid=23412], [object_number=47, tag_tid=8, entry_tid=23411], [object_number=46, tag_tid=8, entry_tid=23410]]], state=connected] 2024-04-24T00:44:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=23413' 2024-04-24T00:44:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=23413' 2024-04-24T00:44:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=23413' '!=' 'object_number=47, tag_tid=8, entry_tid=23295' 2024-04-24T00:44:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=23413' = 'object_number=47, tag_tid=8, entry_tid=23295' 2024-04-24T00:44:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=23413' 2024-04-24T00:44:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=23413' 2024-04-24T00:44:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=23413' 2024-04-24T00:44:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=23413 2024-04-24T00:44:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 23413 2024-04-24T00:44:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=23538], [object_number=45, tag_tid=8, entry_tid=23537], [object_number=44, tag_tid=8, entry_tid=23536], [object_number=47, tag_tid=8, entry_tid=23535]]], state=connected] 2024-04-24T00:44:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=23538], [object_number=45, tag_tid=8, entry_tid=23537], [object_number=44, tag_tid=8, entry_tid=23536], [object_number=47, tag_tid=8, entry_tid=23535]]], state=connected] 2024-04-24T00:44:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=23538' 2024-04-24T00:44:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=23538' 2024-04-24T00:44:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=23538' '!=' 'object_number=45, tag_tid=8, entry_tid=23413' 2024-04-24T00:44:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=23538' = 'object_number=45, tag_tid=8, entry_tid=23413' 2024-04-24T00:44:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=23538' 2024-04-24T00:44:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=23538' 2024-04-24T00:44:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=23538' 2024-04-24T00:44:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=23538 2024-04-24T00:44:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 23538 2024-04-24T00:44:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=23653], [object_number=44, tag_tid=8, entry_tid=23652], [object_number=47, tag_tid=8, entry_tid=23651], [object_number=46, tag_tid=8, entry_tid=23650]]], state=connected] 2024-04-24T00:44:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=23653], [object_number=44, tag_tid=8, entry_tid=23652], [object_number=47, tag_tid=8, entry_tid=23651], [object_number=46, tag_tid=8, entry_tid=23650]]], state=connected] 2024-04-24T00:44:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=23653' 2024-04-24T00:44:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=23653' 2024-04-24T00:44:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=23653' '!=' 'object_number=46, tag_tid=8, entry_tid=23538' 2024-04-24T00:44:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=23653' = 'object_number=46, tag_tid=8, entry_tid=23538' 2024-04-24T00:44:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=23653' 2024-04-24T00:44:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=23653' 2024-04-24T00:44:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=23653' 2024-04-24T00:44:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=23653 2024-04-24T00:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 23653 2024-04-24T00:44:10.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:10.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:10.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23783], [object_number=46, tag_tid=8, entry_tid=23782], [object_number=45, tag_tid=8, entry_tid=23781], [object_number=44, tag_tid=8, entry_tid=23780]]], state=connected] 2024-04-24T00:44:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23783], [object_number=46, tag_tid=8, entry_tid=23782], [object_number=45, tag_tid=8, entry_tid=23781], [object_number=44, tag_tid=8, entry_tid=23780]]], state=connected] 2024-04-24T00:44:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=23783' 2024-04-24T00:44:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=23783' 2024-04-24T00:44:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23783' '!=' 'object_number=45, tag_tid=8, entry_tid=23653' 2024-04-24T00:44:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23783' = 'object_number=45, tag_tid=8, entry_tid=23653' 2024-04-24T00:44:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=23783' 2024-04-24T00:44:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23783' 2024-04-24T00:44:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23783' 2024-04-24T00:44:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=23783 2024-04-24T00:44:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 23783 2024-04-24T00:44:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23911], [object_number=46, tag_tid=8, entry_tid=23910], [object_number=45, tag_tid=8, entry_tid=23909], [object_number=44, tag_tid=8, entry_tid=23908]]], state=connected] 2024-04-24T00:44:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=23911], [object_number=46, tag_tid=8, entry_tid=23910], [object_number=45, tag_tid=8, entry_tid=23909], [object_number=44, tag_tid=8, entry_tid=23908]]], state=connected] 2024-04-24T00:44:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=23911' 2024-04-24T00:44:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=23911' 2024-04-24T00:44:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23911' '!=' 'object_number=47, tag_tid=8, entry_tid=23783' 2024-04-24T00:44:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=23911' = 'object_number=47, tag_tid=8, entry_tid=23783' 2024-04-24T00:44:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=23911' 2024-04-24T00:44:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23911' 2024-04-24T00:44:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=23911' 2024-04-24T00:44:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=23911 2024-04-24T00:44:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 23911 2024-04-24T00:44:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24016], [object_number=47, tag_tid=8, entry_tid=24015], [object_number=46, tag_tid=8, entry_tid=24014], [object_number=45, tag_tid=8, entry_tid=24013]]], state=connected] 2024-04-24T00:44:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24016], [object_number=47, tag_tid=8, entry_tid=24015], [object_number=46, tag_tid=8, entry_tid=24014], [object_number=45, tag_tid=8, entry_tid=24013]]], state=connected] 2024-04-24T00:44:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=24016' 2024-04-24T00:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=24016' 2024-04-24T00:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24016' '!=' 'object_number=47, tag_tid=8, entry_tid=23911' 2024-04-24T00:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24016' = 'object_number=47, tag_tid=8, entry_tid=23911' 2024-04-24T00:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=24016' 2024-04-24T00:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24016' 2024-04-24T00:44:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24016' 2024-04-24T00:44:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=24016 2024-04-24T00:44:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 24016 2024-04-24T00:44:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:12.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=24121], [object_number=44, tag_tid=8, entry_tid=24120], [object_number=47, tag_tid=8, entry_tid=24119], [object_number=46, tag_tid=8, entry_tid=24118]]], state=connected] 2024-04-24T00:44:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=24121], [object_number=44, tag_tid=8, entry_tid=24120], [object_number=47, tag_tid=8, entry_tid=24119], [object_number=46, tag_tid=8, entry_tid=24118]]], state=connected] 2024-04-24T00:44:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=24121' 2024-04-24T00:44:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=24121' 2024-04-24T00:44:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=24121' '!=' 'object_number=44, tag_tid=8, entry_tid=24016' 2024-04-24T00:44:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=24121' = 'object_number=44, tag_tid=8, entry_tid=24016' 2024-04-24T00:44:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=24121' 2024-04-24T00:44:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=24121' 2024-04-24T00:44:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=24121' 2024-04-24T00:44:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=24121 2024-04-24T00:44:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 24121 2024-04-24T00:44:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24224], [object_number=47, tag_tid=8, entry_tid=24223], [object_number=46, tag_tid=8, entry_tid=24222], [object_number=45, tag_tid=8, entry_tid=24221]]], state=connected] 2024-04-24T00:44:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24224], [object_number=47, tag_tid=8, entry_tid=24223], [object_number=46, tag_tid=8, entry_tid=24222], [object_number=45, tag_tid=8, entry_tid=24221]]], state=connected] 2024-04-24T00:44:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=24224' 2024-04-24T00:44:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=24224' 2024-04-24T00:44:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24224' '!=' 'object_number=45, tag_tid=8, entry_tid=24121' 2024-04-24T00:44:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24224' = 'object_number=45, tag_tid=8, entry_tid=24121' 2024-04-24T00:44:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=24224' 2024-04-24T00:44:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24224' 2024-04-24T00:44:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24224' 2024-04-24T00:44:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=24224 2024-04-24T00:44:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 24224 2024-04-24T00:44:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=24337], [object_number=44, tag_tid=8, entry_tid=24336], [object_number=47, tag_tid=8, entry_tid=24335], [object_number=46, tag_tid=8, entry_tid=24334]]], state=connected] 2024-04-24T00:44:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=24337], [object_number=44, tag_tid=8, entry_tid=24336], [object_number=47, tag_tid=8, entry_tid=24335], [object_number=46, tag_tid=8, entry_tid=24334]]], state=connected] 2024-04-24T00:44:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=24337' 2024-04-24T00:44:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=24337' 2024-04-24T00:44:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=24337' '!=' 'object_number=44, tag_tid=8, entry_tid=24224' 2024-04-24T00:44:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=24337' = 'object_number=44, tag_tid=8, entry_tid=24224' 2024-04-24T00:44:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=24337' 2024-04-24T00:44:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=24337' 2024-04-24T00:44:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=24337' 2024-04-24T00:44:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=24337 2024-04-24T00:44:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 24337 2024-04-24T00:44:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=24451], [object_number=46, tag_tid=8, entry_tid=24450], [object_number=45, tag_tid=8, entry_tid=24449], [object_number=44, tag_tid=8, entry_tid=24448]]], state=connected] 2024-04-24T00:44:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=24451], [object_number=46, tag_tid=8, entry_tid=24450], [object_number=45, tag_tid=8, entry_tid=24449], [object_number=44, tag_tid=8, entry_tid=24448]]], state=connected] 2024-04-24T00:44:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=24451' 2024-04-24T00:44:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=24451' 2024-04-24T00:44:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=24451' '!=' 'object_number=45, tag_tid=8, entry_tid=24337' 2024-04-24T00:44:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=24451' = 'object_number=45, tag_tid=8, entry_tid=24337' 2024-04-24T00:44:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=24451' 2024-04-24T00:44:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=24451' 2024-04-24T00:44:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=24451' 2024-04-24T00:44:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=24451 2024-04-24T00:44:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 24451 2024-04-24T00:44:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=24567], [object_number=46, tag_tid=8, entry_tid=24566], [object_number=45, tag_tid=8, entry_tid=24565], [object_number=44, tag_tid=8, entry_tid=24564]]], state=connected] 2024-04-24T00:44:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=24567], [object_number=46, tag_tid=8, entry_tid=24566], [object_number=45, tag_tid=8, entry_tid=24565], [object_number=44, tag_tid=8, entry_tid=24564]]], state=connected] 2024-04-24T00:44:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=24567' 2024-04-24T00:44:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=24567' 2024-04-24T00:44:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=24567' '!=' 'object_number=47, tag_tid=8, entry_tid=24451' 2024-04-24T00:44:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=24567' = 'object_number=47, tag_tid=8, entry_tid=24451' 2024-04-24T00:44:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=24567' 2024-04-24T00:44:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=24567' 2024-04-24T00:44:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=24567' 2024-04-24T00:44:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=24567 2024-04-24T00:44:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 24567 2024-04-24T00:44:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24684], [object_number=47, tag_tid=8, entry_tid=24683], [object_number=46, tag_tid=8, entry_tid=24682], [object_number=45, tag_tid=8, entry_tid=24681]]], state=connected] 2024-04-24T00:44:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24684], [object_number=47, tag_tid=8, entry_tid=24683], [object_number=46, tag_tid=8, entry_tid=24682], [object_number=45, tag_tid=8, entry_tid=24681]]], state=connected] 2024-04-24T00:44:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=24684' 2024-04-24T00:44:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=24684' 2024-04-24T00:44:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24684' '!=' 'object_number=47, tag_tid=8, entry_tid=24567' 2024-04-24T00:44:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24684' = 'object_number=47, tag_tid=8, entry_tid=24567' 2024-04-24T00:44:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=24684' 2024-04-24T00:44:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24684' 2024-04-24T00:44:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24684' 2024-04-24T00:44:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=24684 2024-04-24T00:44:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 24684 2024-04-24T00:44:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24788], [object_number=47, tag_tid=8, entry_tid=24787], [object_number=46, tag_tid=8, entry_tid=24786], [object_number=45, tag_tid=8, entry_tid=24785]]], state=connected] 2024-04-24T00:44:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24788], [object_number=47, tag_tid=8, entry_tid=24787], [object_number=46, tag_tid=8, entry_tid=24786], [object_number=45, tag_tid=8, entry_tid=24785]]], state=connected] 2024-04-24T00:44:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=24788' 2024-04-24T00:44:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=24788' 2024-04-24T00:44:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24788' '!=' 'object_number=44, tag_tid=8, entry_tid=24684' 2024-04-24T00:44:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24788' = 'object_number=44, tag_tid=8, entry_tid=24684' 2024-04-24T00:44:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=24788' 2024-04-24T00:44:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24788' 2024-04-24T00:44:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24788' 2024-04-24T00:44:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=24788 2024-04-24T00:44:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 24788 2024-04-24T00:44:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24928], [object_number=47, tag_tid=8, entry_tid=24927], [object_number=46, tag_tid=8, entry_tid=24926], [object_number=45, tag_tid=8, entry_tid=24925]]], state=connected] 2024-04-24T00:44:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=24928], [object_number=47, tag_tid=8, entry_tid=24927], [object_number=46, tag_tid=8, entry_tid=24926], [object_number=45, tag_tid=8, entry_tid=24925]]], state=connected] 2024-04-24T00:44:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=24928' 2024-04-24T00:44:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=24928' 2024-04-24T00:44:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24928' '!=' 'object_number=44, tag_tid=8, entry_tid=24788' 2024-04-24T00:44:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=24928' = 'object_number=44, tag_tid=8, entry_tid=24788' 2024-04-24T00:44:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=24928' 2024-04-24T00:44:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24928' 2024-04-24T00:44:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=24928' 2024-04-24T00:44:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=24928 2024-04-24T00:44:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 24928 2024-04-24T00:44:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=25069], [object_number=44, tag_tid=8, entry_tid=25068], [object_number=47, tag_tid=8, entry_tid=25067], [object_number=46, tag_tid=8, entry_tid=25066]]], state=connected] 2024-04-24T00:44:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=25069], [object_number=44, tag_tid=8, entry_tid=25068], [object_number=47, tag_tid=8, entry_tid=25067], [object_number=46, tag_tid=8, entry_tid=25066]]], state=connected] 2024-04-24T00:44:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=25069' 2024-04-24T00:44:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=25069' 2024-04-24T00:44:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=25069' '!=' 'object_number=44, tag_tid=8, entry_tid=24928' 2024-04-24T00:44:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=25069' = 'object_number=44, tag_tid=8, entry_tid=24928' 2024-04-24T00:44:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=25069' 2024-04-24T00:44:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=25069' 2024-04-24T00:44:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=25069' 2024-04-24T00:44:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=25069 2024-04-24T00:44:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 25069 2024-04-24T00:44:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=25189], [object_number=44, tag_tid=8, entry_tid=25188], [object_number=47, tag_tid=8, entry_tid=25187], [object_number=46, tag_tid=8, entry_tid=25186]]], state=connected] 2024-04-24T00:44:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=25189], [object_number=44, tag_tid=8, entry_tid=25188], [object_number=47, tag_tid=8, entry_tid=25187], [object_number=46, tag_tid=8, entry_tid=25186]]], state=connected] 2024-04-24T00:44:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=25189' 2024-04-24T00:44:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=25189' 2024-04-24T00:44:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=25189' '!=' 'object_number=45, tag_tid=8, entry_tid=25069' 2024-04-24T00:44:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=25189' = 'object_number=45, tag_tid=8, entry_tid=25069' 2024-04-24T00:44:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=25189' 2024-04-24T00:44:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=25189' 2024-04-24T00:44:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=25189' 2024-04-24T00:44:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=25189 2024-04-24T00:44:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 25189 2024-04-24T00:44:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:18.124 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:44:18.121+0000 7fd951b0e640 -1 reset not still connected to 0x5636f4c85a00 2024-04-24T00:44:18.124 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:44:18.121+0000 7fd951b0e640 -1 reset not still connected to 0x5636f4e511e0 2024-04-24T00:44:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25310], [object_number=45, tag_tid=8, entry_tid=25309], [object_number=44, tag_tid=8, entry_tid=25308], [object_number=47, tag_tid=8, entry_tid=25307]]], state=connected] 2024-04-24T00:44:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25310], [object_number=45, tag_tid=8, entry_tid=25309], [object_number=44, tag_tid=8, entry_tid=25308], [object_number=47, tag_tid=8, entry_tid=25307]]], state=connected] 2024-04-24T00:44:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25310' 2024-04-24T00:44:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25310' 2024-04-24T00:44:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25310' '!=' 'object_number=45, tag_tid=8, entry_tid=25189' 2024-04-24T00:44:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25310' = 'object_number=45, tag_tid=8, entry_tid=25189' 2024-04-24T00:44:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=25310' 2024-04-24T00:44:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25310' 2024-04-24T00:44:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25310' 2024-04-24T00:44:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=25310 2024-04-24T00:44:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 25310 2024-04-24T00:44:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25430], [object_number=45, tag_tid=8, entry_tid=25429], [object_number=44, tag_tid=8, entry_tid=25428], [object_number=47, tag_tid=8, entry_tid=25427]]], state=connected] 2024-04-24T00:44:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25430], [object_number=45, tag_tid=8, entry_tid=25429], [object_number=44, tag_tid=8, entry_tid=25428], [object_number=47, tag_tid=8, entry_tid=25427]]], state=connected] 2024-04-24T00:44:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25430' 2024-04-24T00:44:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25430' 2024-04-24T00:44:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25430' '!=' 'object_number=46, tag_tid=8, entry_tid=25310' 2024-04-24T00:44:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25430' = 'object_number=46, tag_tid=8, entry_tid=25310' 2024-04-24T00:44:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=25430' 2024-04-24T00:44:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25430' 2024-04-24T00:44:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25430' 2024-04-24T00:44:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=25430 2024-04-24T00:44:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 25430 2024-04-24T00:44:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25554], [object_number=45, tag_tid=8, entry_tid=25553], [object_number=44, tag_tid=8, entry_tid=25552], [object_number=47, tag_tid=8, entry_tid=25551]]], state=connected] 2024-04-24T00:44:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25554], [object_number=45, tag_tid=8, entry_tid=25553], [object_number=44, tag_tid=8, entry_tid=25552], [object_number=47, tag_tid=8, entry_tid=25551]]], state=connected] 2024-04-24T00:44:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25554' 2024-04-24T00:44:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25554' 2024-04-24T00:44:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25554' '!=' 'object_number=46, tag_tid=8, entry_tid=25430' 2024-04-24T00:44:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25554' = 'object_number=46, tag_tid=8, entry_tid=25430' 2024-04-24T00:44:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=25554' 2024-04-24T00:44:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25554' 2024-04-24T00:44:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25554' 2024-04-24T00:44:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=25554 2024-04-24T00:44:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 25554 2024-04-24T00:44:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25674], [object_number=45, tag_tid=8, entry_tid=25673], [object_number=44, tag_tid=8, entry_tid=25672], [object_number=47, tag_tid=8, entry_tid=25671]]], state=connected] 2024-04-24T00:44:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25674], [object_number=45, tag_tid=8, entry_tid=25673], [object_number=44, tag_tid=8, entry_tid=25672], [object_number=47, tag_tid=8, entry_tid=25671]]], state=connected] 2024-04-24T00:44:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25674' 2024-04-24T00:44:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25674' 2024-04-24T00:44:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25674' '!=' 'object_number=46, tag_tid=8, entry_tid=25554' 2024-04-24T00:44:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25674' = 'object_number=46, tag_tid=8, entry_tid=25554' 2024-04-24T00:44:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=25674' 2024-04-24T00:44:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25674' 2024-04-24T00:44:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25674' 2024-04-24T00:44:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=25674 2024-04-24T00:44:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 25674 2024-04-24T00:44:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25794], [object_number=45, tag_tid=8, entry_tid=25793], [object_number=44, tag_tid=8, entry_tid=25792], [object_number=47, tag_tid=8, entry_tid=25791]]], state=connected] 2024-04-24T00:44:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25794], [object_number=45, tag_tid=8, entry_tid=25793], [object_number=44, tag_tid=8, entry_tid=25792], [object_number=47, tag_tid=8, entry_tid=25791]]], state=connected] 2024-04-24T00:44:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25794' 2024-04-24T00:44:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25794' 2024-04-24T00:44:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25794' '!=' 'object_number=46, tag_tid=8, entry_tid=25674' 2024-04-24T00:44:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25794' = 'object_number=46, tag_tid=8, entry_tid=25674' 2024-04-24T00:44:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=25794' 2024-04-24T00:44:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25794' 2024-04-24T00:44:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25794' 2024-04-24T00:44:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=25794 2024-04-24T00:44:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 25794 2024-04-24T00:44:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=25915], [object_number=46, tag_tid=8, entry_tid=25914], [object_number=45, tag_tid=8, entry_tid=25913], [object_number=44, tag_tid=8, entry_tid=25912]]], state=connected] 2024-04-24T00:44:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=25915], [object_number=46, tag_tid=8, entry_tid=25914], [object_number=45, tag_tid=8, entry_tid=25913], [object_number=44, tag_tid=8, entry_tid=25912]]], state=connected] 2024-04-24T00:44:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=25915' 2024-04-24T00:44:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=25915' 2024-04-24T00:44:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=25915' '!=' 'object_number=46, tag_tid=8, entry_tid=25794' 2024-04-24T00:44:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=25915' = 'object_number=46, tag_tid=8, entry_tid=25794' 2024-04-24T00:44:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=25915' 2024-04-24T00:44:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=25915' 2024-04-24T00:44:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=25915' 2024-04-24T00:44:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=25915 2024-04-24T00:44:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 25915 2024-04-24T00:44:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25954' '!=' 'object_number=47, tag_tid=8, entry_tid=25915' 2024-04-24T00:44:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:44:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25954' = 'object_number=47, tag_tid=8, entry_tid=25915' 2024-04-24T00:44:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=9 2024-04-24T00:44:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:44:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=8 2024-04-24T00:44:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:44:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:44:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=25954 2024-04-24T00:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 8 -a 1 -le 25954 2024-04-24T00:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:44:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:44:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25954' '!=' 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T00:44:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25954' '!=' 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T00:44:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25954' '!=' 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=25954], [object_number=45, tag_tid=8, entry_tid=25953], [object_number=44, tag_tid=8, entry_tid=25952], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=25954' '!=' 'object_number=46, tag_tid=8, entry_tid=25954' 2024-04-24T00:44:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:44:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T00:44:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:44:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:44:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:44:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:44:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:44:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:44:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:44:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:44:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:44:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:44:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:44:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:44:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:44:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:44:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:44:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:44:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:44:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:44:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:44:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:44:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:44:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:44:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:44:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:44:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:44:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:44:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:44:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:44:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:44:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:44:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:44:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:44:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:44:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:44:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:44:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:44:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:44:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:44:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:44:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:44:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:44:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:44:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:44:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:44:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:44:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:44:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:44:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:44:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:44:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:44:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 11 2024-04-24T00:44:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 11 2024-04-24T00:44:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:44:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=25954], [object_number=47, tag_tid=8, entry_tid=25951]]], state=connected] 2024-04-24T00:44:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:44:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=9, entry_tid=1' 2024-04-24T00:44:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-04-24T00:44:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3 2024-04-24T00:44:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:44:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:44:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:44:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-04-24T00:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 5 2024-04-24T00:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:44:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:42:00 2024 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:43:32 2024 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:44:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:44:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap2-clean ']' 2024-04-24T00:44:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-24T00:44:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2-clean 2024-04-24T00:44:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:44:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:44:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:44:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:44:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:31.507 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:44:31.505+0000 7efd504cb640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:44:31.507 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:44:31.505+0000 7efd4fcca640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:44:31.507 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:44:31.505+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:44:31.508 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:44:31.505+0000 7efd524cf640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:44:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:44:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:32.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap3 2024-04-24T00:44:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3 2024-04-24T00:44:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:44:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:44:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:44:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:44:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:33.349 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:44:33.345+0000 7efd4ccc4640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:44:33.350 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:44:33.349+0000 7efd4f4c9640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:44:33.463 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:44:33.461+0000 7efd504cb640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:44:33.467 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:44:33.465+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:44:33.468 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:44:33.465+0000 7efd4fcca640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:44:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:44:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:44:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap3-clean 2024-04-24T00:44:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-04-24T00:44:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:44:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3-clean 2024-04-24T00:44:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:44:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:44:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:44:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:44:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:44:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T00:44:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:44:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:44:36.687 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5412455057613657) > weight (0.25) 2024-04-24T00:44:36.687 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T00:44:36.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T00:44:36.688 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T00:44:36.688 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T00:44:36.688 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:44:36.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T00:44:36.685+0000 7f81cee91640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 30932) UID: 1000 2024-04-24T00:44:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 42433536 bytes 2024-04-24T00:44:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 110068736 bytes 2024-04-24T00:44:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 130289664 bytes 2024-04-24T00:44:42.789 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T00:44:42.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T00:44:42.789 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T00:44:42.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T00:44:42.792 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-04-24T00:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 172049408 bytes 2024-04-24T00:44:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 244672512 bytes 2024-04-24T00:44:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 268520448 bytes 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:44:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:44:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:44:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:44:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:44:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:44:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:44:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:44:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:44:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:44:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:44:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:44:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:44:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:44:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:44:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":2845088.93,"entries_behind_primary":1,"entries_per_second":178.23,"non_primary_position":{"entry_tid":25954,"object_number":46,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":45,"tag_tid":9},"seconds_until_synced":0} 2024-04-24T00:44:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T00:44:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:44:26 2024-04-24T00:44:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:44:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:44:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:44:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:44:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:44:26' 2024-04-24T00:44:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:44:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:44:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:44:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":2845088.93,"entries_behind_primary":1,"entries_per_second":178.23,"non_primary_position":{"entry_tid":25954,"object_number":46,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":45,"tag_tid":9},"seconds_until_synced":0} 2024-04-24T00:44:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:44:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:44:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:44:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:44:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:44:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T00:44:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:44:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:44:26' 2024-04-24T00:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:44:26' 2024-04-24T00:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:44:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:44:26' 2024-04-24T00:44:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:44:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:44:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:44:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142114271mirror.1smithi13419.0.0-2424-gb98aab6ctrueOK42194274mirror.3smithi13419.0.0-2424-gb98aab6cfalseOK42594312mirror.2smithi13419.0.0-2424-gb98aab6cfalseOK43794382mirror.0smithi13419.0.0-2424-gb98aab6cfalseOKtestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":2845088.93,"entries_behind_primary":1,"entries_per_second":178.23,"non_primary_position":{"entry_tid":25954,"object_number":46,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":45,"tag_tid":9},"seconds_until_synced":0}42114271mirror.1smithi1342024-04-24 00:44:26cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:44:26' 2024-04-24T00:44:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:44:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:44:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:44:26' 2024-04-24T00:44:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:44:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:44:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:44:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:44:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:44:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:44:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4 2024-04-24T00:44:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap4 2024-04-24T00:44:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:44:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:44:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:44:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4 2024-04-24T00:44:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:45:08.415 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:45:08.413+0000 7fbc3ce8e640 -1 reset not still connected to 0x556da9f08820 2024-04-24T00:45:08.415 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:45:08.413+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daeae7d40 2024-04-24T00:45:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:45:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:45:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:45:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:45:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:45:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:45:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:45:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:45:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:45:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:45:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:45:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:45:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:45:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:45:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:45:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:45:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:45:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:45:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:45:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:45:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:45:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:45:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:45:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:45:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:45:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:45:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:45:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:45:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:45:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:45:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:45:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:45:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:45:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=53, tag_tid=11, entry_tid=7805], [object_number=52, tag_tid=11, entry_tid=7804], [object_number=55, tag_tid=11, entry_tid=7803], [object_number=54, tag_tid=11, entry_tid=7802]]], state=connected] 2024-04-24T00:45:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=53, tag_tid=11, entry_tid=7805], [object_number=52, tag_tid=11, entry_tid=7804], [object_number=55, tag_tid=11, entry_tid=7803], [object_number=54, tag_tid=11, entry_tid=7802]]], state=connected] 2024-04-24T00:45:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=53, tag_tid=11, entry_tid=7805' 2024-04-24T00:45:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=53, tag_tid=11, entry_tid=7805' 2024-04-24T00:45:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=53, tag_tid=11, entry_tid=7805' '!=' '' 2024-04-24T00:45:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=53, tag_tid=11, entry_tid=7805' = '' 2024-04-24T00:45:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=53, tag_tid=11, entry_tid=7805' 2024-04-24T00:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=53, tag_tid=11, entry_tid=7805' 2024-04-24T00:45:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=53, tag_tid=11, entry_tid=7805' 2024-04-24T00:45:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7805 2024-04-24T00:45:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 7805 2024-04-24T00:45:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=52, tag_tid=11, entry_tid=7920], [object_number=55, tag_tid=11, entry_tid=7919], [object_number=54, tag_tid=11, entry_tid=7918], [object_number=53, tag_tid=11, entry_tid=7917]]], state=connected] 2024-04-24T00:45:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=52, tag_tid=11, entry_tid=7920], [object_number=55, tag_tid=11, entry_tid=7919], [object_number=54, tag_tid=11, entry_tid=7918], [object_number=53, tag_tid=11, entry_tid=7917]]], state=connected] 2024-04-24T00:45:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=52, tag_tid=11, entry_tid=7920' 2024-04-24T00:45:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=52, tag_tid=11, entry_tid=7920' 2024-04-24T00:45:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=52, tag_tid=11, entry_tid=7920' '!=' 'object_number=53, tag_tid=11, entry_tid=7805' 2024-04-24T00:45:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=52, tag_tid=11, entry_tid=7920' = 'object_number=53, tag_tid=11, entry_tid=7805' 2024-04-24T00:45:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=52, tag_tid=11, entry_tid=7920' 2024-04-24T00:45:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=52, tag_tid=11, entry_tid=7920' 2024-04-24T00:45:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=52, tag_tid=11, entry_tid=7920' 2024-04-24T00:45:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7920 2024-04-24T00:45:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 7920 2024-04-24T00:45:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=53, tag_tid=11, entry_tid=8065], [object_number=52, tag_tid=11, entry_tid=8064], [object_number=55, tag_tid=11, entry_tid=8063], [object_number=54, tag_tid=11, entry_tid=8062]]], state=connected] 2024-04-24T00:45:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=53, tag_tid=11, entry_tid=8065], [object_number=52, tag_tid=11, entry_tid=8064], [object_number=55, tag_tid=11, entry_tid=8063], [object_number=54, tag_tid=11, entry_tid=8062]]], state=connected] 2024-04-24T00:45:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=53, tag_tid=11, entry_tid=8065' 2024-04-24T00:45:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=53, tag_tid=11, entry_tid=8065' 2024-04-24T00:45:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=53, tag_tid=11, entry_tid=8065' '!=' 'object_number=52, tag_tid=11, entry_tid=7920' 2024-04-24T00:45:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=53, tag_tid=11, entry_tid=8065' = 'object_number=52, tag_tid=11, entry_tid=7920' 2024-04-24T00:45:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=53, tag_tid=11, entry_tid=8065' 2024-04-24T00:45:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=53, tag_tid=11, entry_tid=8065' 2024-04-24T00:45:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=53, tag_tid=11, entry_tid=8065' 2024-04-24T00:45:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8065 2024-04-24T00:45:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 8065 2024-04-24T00:45:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=53, tag_tid=11, entry_tid=8169], [object_number=52, tag_tid=11, entry_tid=8168], [object_number=55, tag_tid=11, entry_tid=8167], [object_number=54, tag_tid=11, entry_tid=8166]]], state=connected] 2024-04-24T00:45:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=53, tag_tid=11, entry_tid=8169], [object_number=52, tag_tid=11, entry_tid=8168], [object_number=55, tag_tid=11, entry_tid=8167], [object_number=54, tag_tid=11, entry_tid=8166]]], state=connected] 2024-04-24T00:45:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=53, tag_tid=11, entry_tid=8169' 2024-04-24T00:45:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=53, tag_tid=11, entry_tid=8169' 2024-04-24T00:45:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=53, tag_tid=11, entry_tid=8169' '!=' 'object_number=53, tag_tid=11, entry_tid=8065' 2024-04-24T00:45:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=53, tag_tid=11, entry_tid=8169' = 'object_number=53, tag_tid=11, entry_tid=8065' 2024-04-24T00:45:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=53, tag_tid=11, entry_tid=8169' 2024-04-24T00:45:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=53, tag_tid=11, entry_tid=8169' 2024-04-24T00:45:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=53, tag_tid=11, entry_tid=8169' 2024-04-24T00:45:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8169 2024-04-24T00:45:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 8169 2024-04-24T00:45:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8275], [object_number=54, tag_tid=11, entry_tid=8274], [object_number=53, tag_tid=11, entry_tid=8273], [object_number=52, tag_tid=11, entry_tid=8272]]], state=connected] 2024-04-24T00:45:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8275], [object_number=54, tag_tid=11, entry_tid=8274], [object_number=53, tag_tid=11, entry_tid=8273], [object_number=52, tag_tid=11, entry_tid=8272]]], state=connected] 2024-04-24T00:45:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=55, tag_tid=11, entry_tid=8275' 2024-04-24T00:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=55, tag_tid=11, entry_tid=8275' 2024-04-24T00:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8275' '!=' 'object_number=53, tag_tid=11, entry_tid=8169' 2024-04-24T00:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8275' = 'object_number=53, tag_tid=11, entry_tid=8169' 2024-04-24T00:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=55, tag_tid=11, entry_tid=8275' 2024-04-24T00:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8275' 2024-04-24T00:45:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8275' 2024-04-24T00:45:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8275 2024-04-24T00:45:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 8275 2024-04-24T00:45:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8379], [object_number=54, tag_tid=11, entry_tid=8378], [object_number=53, tag_tid=11, entry_tid=8377], [object_number=52, tag_tid=11, entry_tid=8376]]], state=connected] 2024-04-24T00:45:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8379], [object_number=54, tag_tid=11, entry_tid=8378], [object_number=53, tag_tid=11, entry_tid=8377], [object_number=52, tag_tid=11, entry_tid=8376]]], state=connected] 2024-04-24T00:45:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=55, tag_tid=11, entry_tid=8379' 2024-04-24T00:45:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=55, tag_tid=11, entry_tid=8379' 2024-04-24T00:45:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8379' '!=' 'object_number=55, tag_tid=11, entry_tid=8275' 2024-04-24T00:45:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8379' = 'object_number=55, tag_tid=11, entry_tid=8275' 2024-04-24T00:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=55, tag_tid=11, entry_tid=8379' 2024-04-24T00:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8379' 2024-04-24T00:45:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8379' 2024-04-24T00:45:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8379 2024-04-24T00:45:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 8379 2024-04-24T00:45:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8487], [object_number=54, tag_tid=11, entry_tid=8486], [object_number=53, tag_tid=11, entry_tid=8485], [object_number=52, tag_tid=11, entry_tid=8484]]], state=connected] 2024-04-24T00:45:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8487], [object_number=54, tag_tid=11, entry_tid=8486], [object_number=53, tag_tid=11, entry_tid=8485], [object_number=52, tag_tid=11, entry_tid=8484]]], state=connected] 2024-04-24T00:45:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=55, tag_tid=11, entry_tid=8487' 2024-04-24T00:45:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=55, tag_tid=11, entry_tid=8487' 2024-04-24T00:45:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8487' '!=' 'object_number=55, tag_tid=11, entry_tid=8379' 2024-04-24T00:45:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8487' = 'object_number=55, tag_tid=11, entry_tid=8379' 2024-04-24T00:45:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=55, tag_tid=11, entry_tid=8487' 2024-04-24T00:45:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8487' 2024-04-24T00:45:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8487' 2024-04-24T00:45:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8487 2024-04-24T00:45:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 8487 2024-04-24T00:45:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8599], [object_number=54, tag_tid=11, entry_tid=8598], [object_number=53, tag_tid=11, entry_tid=8597], [object_number=52, tag_tid=11, entry_tid=8596]]], state=connected] 2024-04-24T00:45:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8599], [object_number=54, tag_tid=11, entry_tid=8598], [object_number=53, tag_tid=11, entry_tid=8597], [object_number=52, tag_tid=11, entry_tid=8596]]], state=connected] 2024-04-24T00:45:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=55, tag_tid=11, entry_tid=8599' 2024-04-24T00:45:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=55, tag_tid=11, entry_tid=8599' 2024-04-24T00:45:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8599' '!=' 'object_number=55, tag_tid=11, entry_tid=8487' 2024-04-24T00:45:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8599' = 'object_number=55, tag_tid=11, entry_tid=8487' 2024-04-24T00:45:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=55, tag_tid=11, entry_tid=8599' 2024-04-24T00:45:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8599' 2024-04-24T00:45:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8599' 2024-04-24T00:45:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8599 2024-04-24T00:45:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 8599 2024-04-24T00:45:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8715], [object_number=54, tag_tid=11, entry_tid=8714], [object_number=53, tag_tid=11, entry_tid=8713], [object_number=52, tag_tid=11, entry_tid=8712]]], state=connected] 2024-04-24T00:45:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=8715], [object_number=54, tag_tid=11, entry_tid=8714], [object_number=53, tag_tid=11, entry_tid=8713], [object_number=52, tag_tid=11, entry_tid=8712]]], state=connected] 2024-04-24T00:45:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=55, tag_tid=11, entry_tid=8715' 2024-04-24T00:45:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=55, tag_tid=11, entry_tid=8715' 2024-04-24T00:45:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8715' '!=' 'object_number=55, tag_tid=11, entry_tid=8599' 2024-04-24T00:45:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=8715' = 'object_number=55, tag_tid=11, entry_tid=8599' 2024-04-24T00:45:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=55, tag_tid=11, entry_tid=8715' 2024-04-24T00:45:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8715' 2024-04-24T00:45:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=8715' 2024-04-24T00:45:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8715 2024-04-24T00:45:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 8715 2024-04-24T00:45:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=52, tag_tid=11, entry_tid=8832], [object_number=55, tag_tid=11, entry_tid=8831], [object_number=54, tag_tid=11, entry_tid=8830], [object_number=53, tag_tid=11, entry_tid=8829]]], state=connected] 2024-04-24T00:45:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 13 2024-04-24T00:45:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=52, tag_tid=11, entry_tid=8832], [object_number=55, tag_tid=11, entry_tid=8831], [object_number=54, tag_tid=11, entry_tid=8830], [object_number=53, tag_tid=11, entry_tid=8829]]], state=connected] 2024-04-24T00:45:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=52, tag_tid=11, entry_tid=8832' 2024-04-24T00:45:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=52, tag_tid=11, entry_tid=8832' 2024-04-24T00:45:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=52, tag_tid=11, entry_tid=8832' '!=' 'object_number=55, tag_tid=11, entry_tid=8715' 2024-04-24T00:45:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=52, tag_tid=11, entry_tid=8832' = 'object_number=55, tag_tid=11, entry_tid=8715' 2024-04-24T00:45:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=52, tag_tid=11, entry_tid=8832' 2024-04-24T00:45:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=52, tag_tid=11, entry_tid=8832' 2024-04-24T00:45:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=52, tag_tid=11, entry_tid=8832' 2024-04-24T00:45:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8832 2024-04-24T00:45:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 8832 2024-04-24T00:45:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:45:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8952], [object_number=59, tag_tid=11, entry_tid=8951], [object_number=58, tag_tid=11, entry_tid=8950], [object_number=57, tag_tid=11, entry_tid=8949]]], state=connected] 2024-04-24T00:45:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:45:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8952], [object_number=59, tag_tid=11, entry_tid=8951], [object_number=58, tag_tid=11, entry_tid=8950], [object_number=57, tag_tid=11, entry_tid=8949]]], state=connected] 2024-04-24T00:45:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=8952' 2024-04-24T00:45:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=8952' 2024-04-24T00:45:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8952' '!=' 'object_number=52, tag_tid=11, entry_tid=8832' 2024-04-24T00:45:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8952' = 'object_number=52, tag_tid=11, entry_tid=8832' 2024-04-24T00:45:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=8952' 2024-04-24T00:45:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8952' 2024-04-24T00:45:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8952' 2024-04-24T00:45:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8952 2024-04-24T00:45:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 8952 2024-04-24T00:45:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:45:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:45:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:45:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:45:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:45:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=9066], [object_number=57, tag_tid=11, entry_tid=9065], [object_number=56, tag_tid=11, entry_tid=9064], [object_number=59, tag_tid=11, entry_tid=9063]]], state=connected] 2024-04-24T00:45:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:45:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:45:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:45:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:45:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:45:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:45:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:45:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:45:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=9066], [object_number=57, tag_tid=11, entry_tid=9065], [object_number=56, tag_tid=11, entry_tid=9064], [object_number=59, tag_tid=11, entry_tid=9063]]], state=connected] 2024-04-24T00:45:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:45:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=9066' 2024-04-24T00:45:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=9066' 2024-04-24T00:45:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=9066' '!=' 'object_number=56, tag_tid=11, entry_tid=8952' 2024-04-24T00:45:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:45:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=9066' = 'object_number=56, tag_tid=11, entry_tid=8952' 2024-04-24T00:45:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=9066' 2024-04-24T00:45:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:45:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=9066' 2024-04-24T00:45:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:45:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:45:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:45:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:45:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=9066' 2024-04-24T00:45:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:45:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:45:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9066 2024-04-24T00:45:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 9066 2024-04-24T00:45:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:45:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:45:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:45:26.682 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.37461069865295527) > weight (0.25) 2024-04-24T00:45:26.682 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T00:45:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T00:45:26.683 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T00:45:26.683 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T00:45:26.683 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:45:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T00:45:26.685+0000 7f8565cb8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 29649) UID: 1000 2024-04-24T00:45:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:45:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:45:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:45:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:45:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:45:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:45:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:45:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:45:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:45:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:45:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:45:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:26.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:45:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:45:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:45:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:45:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:45:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:45:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:45:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:45:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:45:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:45:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:45:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:45:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:45:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:45:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:45:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:45:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:45:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:45:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:45:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:45:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:45:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:45:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:45:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:45:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T00:45:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:45:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:45:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:45:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:45:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:45:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:45:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:45:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:45:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:45:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:45:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:45:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:45:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:45:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:45:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:45:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:45:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:45:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:45:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:45:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:45:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:45:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:45:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:45:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:45:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:45:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:45:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:45:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:45:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:45:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 4 2024-04-24T00:45:32.785 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T00:45:32.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T00:45:32.786 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T00:45:32.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T00:45:32.789 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-04-24T00:45:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:45:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:45:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:45:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:45:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:45:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:45:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:45:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:45:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:45:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:45:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:45:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:45:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:45:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T00:45:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:45:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:45:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:45:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:45:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:45:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:45:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:45:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:45:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:45:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:45:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:45:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:45:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:45:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:45:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:45:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T00:45:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:45:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:45:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:45:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:45:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:45:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:45:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:45:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:45:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:45:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:45:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:45:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:45:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:45:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:45:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:45:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:45:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:45:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:45:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:45:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:45:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:45:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:45:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:45:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:45:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:45:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:45:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:45:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:45:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:45:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:45:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:45:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:45:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:45:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:45:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:45:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:45:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:45:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:45:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:45:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:45:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:45:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:45:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:45:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:45:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:45:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:45:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:45:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:45:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T00:46:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=9356], [object_number=59, tag_tid=11, entry_tid=9355], [object_number=58, tag_tid=11, entry_tid=9354], [object_number=57, tag_tid=11, entry_tid=9353]]], state=connected] 2024-04-24T00:46:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=9356], [object_number=59, tag_tid=11, entry_tid=9355], [object_number=58, tag_tid=11, entry_tid=9354], [object_number=57, tag_tid=11, entry_tid=9353]]], state=connected] 2024-04-24T00:46:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=9356' 2024-04-24T00:46:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=9356' 2024-04-24T00:46:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=9356' '!=' 'object_number=58, tag_tid=11, entry_tid=9066' 2024-04-24T00:46:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=9356' = 'object_number=58, tag_tid=11, entry_tid=9066' 2024-04-24T00:46:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=9356' 2024-04-24T00:46:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=9356' 2024-04-24T00:46:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=9356' 2024-04-24T00:46:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9356 2024-04-24T00:46:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 9356 2024-04-24T00:46:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9527], [object_number=58, tag_tid=11, entry_tid=9526], [object_number=57, tag_tid=11, entry_tid=9525], [object_number=56, tag_tid=11, entry_tid=9524]]], state=connected] 2024-04-24T00:46:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9527], [object_number=58, tag_tid=11, entry_tid=9526], [object_number=57, tag_tid=11, entry_tid=9525], [object_number=56, tag_tid=11, entry_tid=9524]]], state=connected] 2024-04-24T00:46:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:02.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=9527' 2024-04-24T00:46:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=9527' 2024-04-24T00:46:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9527' '!=' 'object_number=56, tag_tid=11, entry_tid=9356' 2024-04-24T00:46:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9527' = 'object_number=56, tag_tid=11, entry_tid=9356' 2024-04-24T00:46:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=9527' 2024-04-24T00:46:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9527' 2024-04-24T00:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9527' 2024-04-24T00:46:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9527 2024-04-24T00:46:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 9527 2024-04-24T00:46:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9695], [object_number=58, tag_tid=11, entry_tid=9694], [object_number=57, tag_tid=11, entry_tid=9693], [object_number=56, tag_tid=11, entry_tid=9692]]], state=connected] 2024-04-24T00:46:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9695], [object_number=58, tag_tid=11, entry_tid=9694], [object_number=57, tag_tid=11, entry_tid=9693], [object_number=56, tag_tid=11, entry_tid=9692]]], state=connected] 2024-04-24T00:46:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=9695' 2024-04-24T00:46:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=9695' 2024-04-24T00:46:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9695' '!=' 'object_number=59, tag_tid=11, entry_tid=9527' 2024-04-24T00:46:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9695' = 'object_number=59, tag_tid=11, entry_tid=9527' 2024-04-24T00:46:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=9695' 2024-04-24T00:46:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9695' 2024-04-24T00:46:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9695' 2024-04-24T00:46:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9695 2024-04-24T00:46:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 9695 2024-04-24T00:46:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:03.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:03.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:03.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:03.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9851], [object_number=58, tag_tid=11, entry_tid=9850], [object_number=57, tag_tid=11, entry_tid=9849], [object_number=56, tag_tid=11, entry_tid=9848]]], state=connected] 2024-04-24T00:46:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9851], [object_number=58, tag_tid=11, entry_tid=9850], [object_number=57, tag_tid=11, entry_tid=9849], [object_number=56, tag_tid=11, entry_tid=9848]]], state=connected] 2024-04-24T00:46:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=9851' 2024-04-24T00:46:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=9851' 2024-04-24T00:46:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9851' '!=' 'object_number=59, tag_tid=11, entry_tid=9695' 2024-04-24T00:46:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9851' = 'object_number=59, tag_tid=11, entry_tid=9695' 2024-04-24T00:46:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=9851' 2024-04-24T00:46:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9851' 2024-04-24T00:46:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9851' 2024-04-24T00:46:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9851 2024-04-24T00:46:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 9851 2024-04-24T00:46:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=10002], [object_number=57, tag_tid=11, entry_tid=10001], [object_number=56, tag_tid=11, entry_tid=10000], [object_number=59, tag_tid=11, entry_tid=9999]]], state=connected] 2024-04-24T00:46:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=10002], [object_number=57, tag_tid=11, entry_tid=10001], [object_number=56, tag_tid=11, entry_tid=10000], [object_number=59, tag_tid=11, entry_tid=9999]]], state=connected] 2024-04-24T00:46:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=10002' 2024-04-24T00:46:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=10002' 2024-04-24T00:46:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=10002' '!=' 'object_number=59, tag_tid=11, entry_tid=9851' 2024-04-24T00:46:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=10002' = 'object_number=59, tag_tid=11, entry_tid=9851' 2024-04-24T00:46:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=10002' 2024-04-24T00:46:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=10002' 2024-04-24T00:46:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=10002' 2024-04-24T00:46:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10002 2024-04-24T00:46:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 10002 2024-04-24T00:46:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=10137], [object_number=56, tag_tid=11, entry_tid=10136], [object_number=59, tag_tid=11, entry_tid=10135], [object_number=58, tag_tid=11, entry_tid=10134]]], state=connected] 2024-04-24T00:46:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=10137], [object_number=56, tag_tid=11, entry_tid=10136], [object_number=59, tag_tid=11, entry_tid=10135], [object_number=58, tag_tid=11, entry_tid=10134]]], state=connected] 2024-04-24T00:46:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=10137' 2024-04-24T00:46:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=10137' 2024-04-24T00:46:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=10137' '!=' 'object_number=58, tag_tid=11, entry_tid=10002' 2024-04-24T00:46:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=10137' = 'object_number=58, tag_tid=11, entry_tid=10002' 2024-04-24T00:46:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=10137' 2024-04-24T00:46:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=10137' 2024-04-24T00:46:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=10137' 2024-04-24T00:46:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10137 2024-04-24T00:46:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 10137 2024-04-24T00:46:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=10268], [object_number=59, tag_tid=11, entry_tid=10267], [object_number=58, tag_tid=11, entry_tid=10266], [object_number=57, tag_tid=11, entry_tid=10265]]], state=connected] 2024-04-24T00:46:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=10268], [object_number=59, tag_tid=11, entry_tid=10267], [object_number=58, tag_tid=11, entry_tid=10266], [object_number=57, tag_tid=11, entry_tid=10265]]], state=connected] 2024-04-24T00:46:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=10268' 2024-04-24T00:46:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=10268' 2024-04-24T00:46:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=10268' '!=' 'object_number=57, tag_tid=11, entry_tid=10137' 2024-04-24T00:46:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=10268' = 'object_number=57, tag_tid=11, entry_tid=10137' 2024-04-24T00:46:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=10268' 2024-04-24T00:46:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=10268' 2024-04-24T00:46:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=10268' 2024-04-24T00:46:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10268 2024-04-24T00:46:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 10268 2024-04-24T00:46:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=10433], [object_number=56, tag_tid=11, entry_tid=10432], [object_number=59, tag_tid=11, entry_tid=10431], [object_number=58, tag_tid=11, entry_tid=10430]]], state=connected] 2024-04-24T00:46:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=10433], [object_number=56, tag_tid=11, entry_tid=10432], [object_number=59, tag_tid=11, entry_tid=10431], [object_number=58, tag_tid=11, entry_tid=10430]]], state=connected] 2024-04-24T00:46:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=10433' 2024-04-24T00:46:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=10433' 2024-04-24T00:46:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=10433' '!=' 'object_number=56, tag_tid=11, entry_tid=10268' 2024-04-24T00:46:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=10433' = 'object_number=56, tag_tid=11, entry_tid=10268' 2024-04-24T00:46:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=10433' 2024-04-24T00:46:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=10433' 2024-04-24T00:46:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:07.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=10433' 2024-04-24T00:46:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10433 2024-04-24T00:46:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 10433 2024-04-24T00:46:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=10591], [object_number=58, tag_tid=11, entry_tid=10590], [object_number=57, tag_tid=11, entry_tid=10589], [object_number=56, tag_tid=11, entry_tid=10588]]], state=connected] 2024-04-24T00:46:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=10591], [object_number=58, tag_tid=11, entry_tid=10590], [object_number=57, tag_tid=11, entry_tid=10589], [object_number=56, tag_tid=11, entry_tid=10588]]], state=connected] 2024-04-24T00:46:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=10591' 2024-04-24T00:46:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=10591' 2024-04-24T00:46:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=10591' '!=' 'object_number=57, tag_tid=11, entry_tid=10433' 2024-04-24T00:46:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=10591' = 'object_number=57, tag_tid=11, entry_tid=10433' 2024-04-24T00:46:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=10591' 2024-04-24T00:46:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=10591' 2024-04-24T00:46:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=10591' 2024-04-24T00:46:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10591 2024-04-24T00:46:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 10591 2024-04-24T00:46:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=10755], [object_number=58, tag_tid=11, entry_tid=10754], [object_number=57, tag_tid=11, entry_tid=10753], [object_number=56, tag_tid=11, entry_tid=10752]]], state=connected] 2024-04-24T00:46:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=10755], [object_number=58, tag_tid=11, entry_tid=10754], [object_number=57, tag_tid=11, entry_tid=10753], [object_number=56, tag_tid=11, entry_tid=10752]]], state=connected] 2024-04-24T00:46:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=10755' 2024-04-24T00:46:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=10755' 2024-04-24T00:46:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=10755' '!=' 'object_number=59, tag_tid=11, entry_tid=10591' 2024-04-24T00:46:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=10755' = 'object_number=59, tag_tid=11, entry_tid=10591' 2024-04-24T00:46:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=10755' 2024-04-24T00:46:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=10755' 2024-04-24T00:46:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=10755' 2024-04-24T00:46:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10755 2024-04-24T00:46:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 10755 2024-04-24T00:46:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=10932], [object_number=59, tag_tid=11, entry_tid=10931], [object_number=58, tag_tid=11, entry_tid=10930], [object_number=57, tag_tid=11, entry_tid=10929]]], state=connected] 2024-04-24T00:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=10932], [object_number=59, tag_tid=11, entry_tid=10931], [object_number=58, tag_tid=11, entry_tid=10930], [object_number=57, tag_tid=11, entry_tid=10929]]], state=connected] 2024-04-24T00:46:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=10932' 2024-04-24T00:46:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=10932' 2024-04-24T00:46:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=10932' '!=' 'object_number=59, tag_tid=11, entry_tid=10755' 2024-04-24T00:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=10932' = 'object_number=59, tag_tid=11, entry_tid=10755' 2024-04-24T00:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=10932' 2024-04-24T00:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=10932' 2024-04-24T00:46:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=10932' 2024-04-24T00:46:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10932 2024-04-24T00:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 10932 2024-04-24T00:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=11111], [object_number=58, tag_tid=11, entry_tid=11110], [object_number=57, tag_tid=11, entry_tid=11109], [object_number=56, tag_tid=11, entry_tid=11108]]], state=connected] 2024-04-24T00:46:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=11111], [object_number=58, tag_tid=11, entry_tid=11110], [object_number=57, tag_tid=11, entry_tid=11109], [object_number=56, tag_tid=11, entry_tid=11108]]], state=connected] 2024-04-24T00:46:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=11111' 2024-04-24T00:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=11111' 2024-04-24T00:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=11111' '!=' 'object_number=56, tag_tid=11, entry_tid=10932' 2024-04-24T00:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=11111' = 'object_number=56, tag_tid=11, entry_tid=10932' 2024-04-24T00:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=11111' 2024-04-24T00:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=11111' 2024-04-24T00:46:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=11111' 2024-04-24T00:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11111 2024-04-24T00:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 11111 2024-04-24T00:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=11280], [object_number=59, tag_tid=11, entry_tid=11279], [object_number=58, tag_tid=11, entry_tid=11278], [object_number=57, tag_tid=11, entry_tid=11277]]], state=connected] 2024-04-24T00:46:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=11280], [object_number=59, tag_tid=11, entry_tid=11279], [object_number=58, tag_tid=11, entry_tid=11278], [object_number=57, tag_tid=11, entry_tid=11277]]], state=connected] 2024-04-24T00:46:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=11280' 2024-04-24T00:46:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=11280' 2024-04-24T00:46:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=11280' '!=' 'object_number=59, tag_tid=11, entry_tid=11111' 2024-04-24T00:46:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=11280' = 'object_number=59, tag_tid=11, entry_tid=11111' 2024-04-24T00:46:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=11280' 2024-04-24T00:46:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=11280' 2024-04-24T00:46:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=11280' 2024-04-24T00:46:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11280 2024-04-24T00:46:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 11280 2024-04-24T00:46:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=11439], [object_number=58, tag_tid=11, entry_tid=11438], [object_number=57, tag_tid=11, entry_tid=11437], [object_number=56, tag_tid=11, entry_tid=11436]]], state=connected] 2024-04-24T00:46:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=11439], [object_number=58, tag_tid=11, entry_tid=11438], [object_number=57, tag_tid=11, entry_tid=11437], [object_number=56, tag_tid=11, entry_tid=11436]]], state=connected] 2024-04-24T00:46:13.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=11439' 2024-04-24T00:46:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=11439' 2024-04-24T00:46:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=11439' '!=' 'object_number=56, tag_tid=11, entry_tid=11280' 2024-04-24T00:46:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=11439' = 'object_number=56, tag_tid=11, entry_tid=11280' 2024-04-24T00:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=11439' 2024-04-24T00:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=11439' 2024-04-24T00:46:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=11439' 2024-04-24T00:46:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11439 2024-04-24T00:46:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 11439 2024-04-24T00:46:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=11613], [object_number=56, tag_tid=11, entry_tid=11612], [object_number=59, tag_tid=11, entry_tid=11611], [object_number=58, tag_tid=11, entry_tid=11610]]], state=connected] 2024-04-24T00:46:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=11613], [object_number=56, tag_tid=11, entry_tid=11612], [object_number=59, tag_tid=11, entry_tid=11611], [object_number=58, tag_tid=11, entry_tid=11610]]], state=connected] 2024-04-24T00:46:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=11613' 2024-04-24T00:46:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=11613' 2024-04-24T00:46:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=11613' '!=' 'object_number=59, tag_tid=11, entry_tid=11439' 2024-04-24T00:46:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=11613' = 'object_number=59, tag_tid=11, entry_tid=11439' 2024-04-24T00:46:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=11613' 2024-04-24T00:46:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=11613' 2024-04-24T00:46:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=11613' 2024-04-24T00:46:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11613 2024-04-24T00:46:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 11613 2024-04-24T00:46:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:14.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=11788], [object_number=59, tag_tid=11, entry_tid=11787], [object_number=58, tag_tid=11, entry_tid=11786], [object_number=57, tag_tid=11, entry_tid=11785]]], state=connected] 2024-04-24T00:46:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=11788], [object_number=59, tag_tid=11, entry_tid=11787], [object_number=58, tag_tid=11, entry_tid=11786], [object_number=57, tag_tid=11, entry_tid=11785]]], state=connected] 2024-04-24T00:46:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=11788' 2024-04-24T00:46:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=11788' 2024-04-24T00:46:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=11788' '!=' 'object_number=57, tag_tid=11, entry_tid=11613' 2024-04-24T00:46:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=11788' = 'object_number=57, tag_tid=11, entry_tid=11613' 2024-04-24T00:46:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=11788' 2024-04-24T00:46:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=11788' 2024-04-24T00:46:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=11788' 2024-04-24T00:46:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11788 2024-04-24T00:46:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 11788 2024-04-24T00:46:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=11955], [object_number=58, tag_tid=11, entry_tid=11954], [object_number=57, tag_tid=11, entry_tid=11953], [object_number=56, tag_tid=11, entry_tid=11952]]], state=connected] 2024-04-24T00:46:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=11955], [object_number=58, tag_tid=11, entry_tid=11954], [object_number=57, tag_tid=11, entry_tid=11953], [object_number=56, tag_tid=11, entry_tid=11952]]], state=connected] 2024-04-24T00:46:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=11955' 2024-04-24T00:46:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=11955' 2024-04-24T00:46:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=11955' '!=' 'object_number=56, tag_tid=11, entry_tid=11788' 2024-04-24T00:46:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=11955' = 'object_number=56, tag_tid=11, entry_tid=11788' 2024-04-24T00:46:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=11955' 2024-04-24T00:46:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=11955' 2024-04-24T00:46:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=11955' 2024-04-24T00:46:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11955 2024-04-24T00:46:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 11955 2024-04-24T00:46:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=12101], [object_number=56, tag_tid=11, entry_tid=12100], [object_number=59, tag_tid=11, entry_tid=12099], [object_number=58, tag_tid=11, entry_tid=12098]]], state=connected] 2024-04-24T00:46:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=12101], [object_number=56, tag_tid=11, entry_tid=12100], [object_number=59, tag_tid=11, entry_tid=12099], [object_number=58, tag_tid=11, entry_tid=12098]]], state=connected] 2024-04-24T00:46:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=12101' 2024-04-24T00:46:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=12101' 2024-04-24T00:46:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=12101' '!=' 'object_number=59, tag_tid=11, entry_tid=11955' 2024-04-24T00:46:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=12101' = 'object_number=59, tag_tid=11, entry_tid=11955' 2024-04-24T00:46:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=12101' 2024-04-24T00:46:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=12101' 2024-04-24T00:46:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=12101' 2024-04-24T00:46:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12101 2024-04-24T00:46:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 12101 2024-04-24T00:46:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=12261], [object_number=56, tag_tid=11, entry_tid=12260], [object_number=59, tag_tid=11, entry_tid=12259], [object_number=58, tag_tid=11, entry_tid=12258]]], state=connected] 2024-04-24T00:46:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=12261], [object_number=56, tag_tid=11, entry_tid=12260], [object_number=59, tag_tid=11, entry_tid=12259], [object_number=58, tag_tid=11, entry_tid=12258]]], state=connected] 2024-04-24T00:46:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=12261' 2024-04-24T00:46:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=12261' 2024-04-24T00:46:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=12261' '!=' 'object_number=57, tag_tid=11, entry_tid=12101' 2024-04-24T00:46:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=12261' = 'object_number=57, tag_tid=11, entry_tid=12101' 2024-04-24T00:46:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=12261' 2024-04-24T00:46:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=12261' 2024-04-24T00:46:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=12261' 2024-04-24T00:46:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12261 2024-04-24T00:46:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 12261 2024-04-24T00:46:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=12427], [object_number=58, tag_tid=11, entry_tid=12426], [object_number=57, tag_tid=11, entry_tid=12425], [object_number=56, tag_tid=11, entry_tid=12424]]], state=connected] 2024-04-24T00:46:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=12427], [object_number=58, tag_tid=11, entry_tid=12426], [object_number=57, tag_tid=11, entry_tid=12425], [object_number=56, tag_tid=11, entry_tid=12424]]], state=connected] 2024-04-24T00:46:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=12427' 2024-04-24T00:46:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=12427' 2024-04-24T00:46:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=12427' '!=' 'object_number=57, tag_tid=11, entry_tid=12261' 2024-04-24T00:46:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=12427' = 'object_number=57, tag_tid=11, entry_tid=12261' 2024-04-24T00:46:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=12427' 2024-04-24T00:46:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=12427' 2024-04-24T00:46:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=12427' 2024-04-24T00:46:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12427 2024-04-24T00:46:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 12427 2024-04-24T00:46:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=12584], [object_number=59, tag_tid=11, entry_tid=12583], [object_number=58, tag_tid=11, entry_tid=12582], [object_number=57, tag_tid=11, entry_tid=12581]]], state=connected] 2024-04-24T00:46:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=12584], [object_number=59, tag_tid=11, entry_tid=12583], [object_number=58, tag_tid=11, entry_tid=12582], [object_number=57, tag_tid=11, entry_tid=12581]]], state=connected] 2024-04-24T00:46:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=12584' 2024-04-24T00:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=12584' 2024-04-24T00:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=12584' '!=' 'object_number=59, tag_tid=11, entry_tid=12427' 2024-04-24T00:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=12584' = 'object_number=59, tag_tid=11, entry_tid=12427' 2024-04-24T00:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=12584' 2024-04-24T00:46:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=12584' 2024-04-24T00:46:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=12584' 2024-04-24T00:46:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12584 2024-04-24T00:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 12584 2024-04-24T00:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=12761], [object_number=56, tag_tid=11, entry_tid=12760], [object_number=59, tag_tid=11, entry_tid=12759], [object_number=58, tag_tid=11, entry_tid=12758]]], state=connected] 2024-04-24T00:46:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=12761], [object_number=56, tag_tid=11, entry_tid=12760], [object_number=59, tag_tid=11, entry_tid=12759], [object_number=58, tag_tid=11, entry_tid=12758]]], state=connected] 2024-04-24T00:46:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=12761' 2024-04-24T00:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=12761' 2024-04-24T00:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=12761' '!=' 'object_number=56, tag_tid=11, entry_tid=12584' 2024-04-24T00:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=12761' = 'object_number=56, tag_tid=11, entry_tid=12584' 2024-04-24T00:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=12761' 2024-04-24T00:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=12761' 2024-04-24T00:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=12761' 2024-04-24T00:46:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12761 2024-04-24T00:46:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 12761 2024-04-24T00:46:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:20.793 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T00:46:20.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T00:46:20.793 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T00:46:20.793 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T00:46:20.793 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:46:20.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T00:46:20.792+0000 7f30d9f15640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 35907) UID: 1000 2024-04-24T00:46:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=12987], [object_number=58, tag_tid=11, entry_tid=12986], [object_number=57, tag_tid=11, entry_tid=12985], [object_number=56, tag_tid=11, entry_tid=12984]]], state=connected] 2024-04-24T00:46:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 14 2024-04-24T00:46:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=12987], [object_number=58, tag_tid=11, entry_tid=12986], [object_number=57, tag_tid=11, entry_tid=12985], [object_number=56, tag_tid=11, entry_tid=12984]]], state=connected] 2024-04-24T00:46:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=12987' 2024-04-24T00:46:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=12987' 2024-04-24T00:46:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=12987' '!=' 'object_number=57, tag_tid=11, entry_tid=12761' 2024-04-24T00:46:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=12987' = 'object_number=57, tag_tid=11, entry_tid=12761' 2024-04-24T00:46:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=12987' 2024-04-24T00:46:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=12987' 2024-04-24T00:46:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=12987' 2024-04-24T00:46:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12987 2024-04-24T00:46:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 12987 2024-04-24T00:46:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=13132], [object_number=63, tag_tid=11, entry_tid=13131], [object_number=62, tag_tid=11, entry_tid=13130], [object_number=61, tag_tid=11, entry_tid=13129]]], state=connected] 2024-04-24T00:46:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=13132], [object_number=63, tag_tid=11, entry_tid=13131], [object_number=62, tag_tid=11, entry_tid=13130], [object_number=61, tag_tid=11, entry_tid=13129]]], state=connected] 2024-04-24T00:46:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=13132' 2024-04-24T00:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=13132' 2024-04-24T00:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=13132' '!=' 'object_number=59, tag_tid=11, entry_tid=12987' 2024-04-24T00:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=13132' = 'object_number=59, tag_tid=11, entry_tid=12987' 2024-04-24T00:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=13132' 2024-04-24T00:46:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=13132' 2024-04-24T00:46:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=13132' 2024-04-24T00:46:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13132 2024-04-24T00:46:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 13132 2024-04-24T00:46:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=13306], [object_number=61, tag_tid=11, entry_tid=13305], [object_number=60, tag_tid=11, entry_tid=13304], [object_number=63, tag_tid=11, entry_tid=13303]]], state=connected] 2024-04-24T00:46:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=13306], [object_number=61, tag_tid=11, entry_tid=13305], [object_number=60, tag_tid=11, entry_tid=13304], [object_number=63, tag_tid=11, entry_tid=13303]]], state=connected] 2024-04-24T00:46:22.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=13306' 2024-04-24T00:46:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=13306' 2024-04-24T00:46:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=13306' '!=' 'object_number=60, tag_tid=11, entry_tid=13132' 2024-04-24T00:46:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=13306' = 'object_number=60, tag_tid=11, entry_tid=13132' 2024-04-24T00:46:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=13306' 2024-04-24T00:46:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=13306' 2024-04-24T00:46:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=13306' 2024-04-24T00:46:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13306 2024-04-24T00:46:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 13306 2024-04-24T00:46:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=13482], [object_number=61, tag_tid=11, entry_tid=13481], [object_number=60, tag_tid=11, entry_tid=13480], [object_number=63, tag_tid=11, entry_tid=13479]]], state=connected] 2024-04-24T00:46:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=13482], [object_number=61, tag_tid=11, entry_tid=13481], [object_number=60, tag_tid=11, entry_tid=13480], [object_number=63, tag_tid=11, entry_tid=13479]]], state=connected] 2024-04-24T00:46:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=13482' 2024-04-24T00:46:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=13482' 2024-04-24T00:46:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=13482' '!=' 'object_number=62, tag_tid=11, entry_tid=13306' 2024-04-24T00:46:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=13482' = 'object_number=62, tag_tid=11, entry_tid=13306' 2024-04-24T00:46:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=13482' 2024-04-24T00:46:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=13482' 2024-04-24T00:46:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=13482' 2024-04-24T00:46:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13482 2024-04-24T00:46:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 13482 2024-04-24T00:46:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13737], [object_number=60, tag_tid=11, entry_tid=13736], [object_number=63, tag_tid=11, entry_tid=13735], [object_number=62, tag_tid=11, entry_tid=13734]]], state=connected] 2024-04-24T00:46:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13737], [object_number=60, tag_tid=11, entry_tid=13736], [object_number=63, tag_tid=11, entry_tid=13735], [object_number=62, tag_tid=11, entry_tid=13734]]], state=connected] 2024-04-24T00:46:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=13737' 2024-04-24T00:46:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=13737' 2024-04-24T00:46:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13737' '!=' 'object_number=62, tag_tid=11, entry_tid=13482' 2024-04-24T00:46:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13737' = 'object_number=62, tag_tid=11, entry_tid=13482' 2024-04-24T00:46:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=13737' 2024-04-24T00:46:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13737' 2024-04-24T00:46:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13737' 2024-04-24T00:46:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13737 2024-04-24T00:46:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 13737 2024-04-24T00:46:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=13919], [object_number=62, tag_tid=11, entry_tid=13918], [object_number=61, tag_tid=11, entry_tid=13917], [object_number=60, tag_tid=11, entry_tid=13916]]], state=connected] 2024-04-24T00:46:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=13919], [object_number=62, tag_tid=11, entry_tid=13918], [object_number=61, tag_tid=11, entry_tid=13917], [object_number=60, tag_tid=11, entry_tid=13916]]], state=connected] 2024-04-24T00:46:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=13919' 2024-04-24T00:46:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=13919' 2024-04-24T00:46:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=13919' '!=' 'object_number=61, tag_tid=11, entry_tid=13737' 2024-04-24T00:46:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=13919' = 'object_number=61, tag_tid=11, entry_tid=13737' 2024-04-24T00:46:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=13919' 2024-04-24T00:46:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=13919' 2024-04-24T00:46:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=13919' 2024-04-24T00:46:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13919 2024-04-24T00:46:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 13919 2024-04-24T00:46:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:26.895 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T00:46:26.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T00:46:26.896 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T00:46:26.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T00:46:26.899 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-04-24T00:46:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14091], [object_number=62, tag_tid=11, entry_tid=14090], [object_number=61, tag_tid=11, entry_tid=14089], [object_number=60, tag_tid=11, entry_tid=14088]]], state=connected] 2024-04-24T00:46:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14091], [object_number=62, tag_tid=11, entry_tid=14090], [object_number=61, tag_tid=11, entry_tid=14089], [object_number=60, tag_tid=11, entry_tid=14088]]], state=connected] 2024-04-24T00:46:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=14091' 2024-04-24T00:46:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=14091' 2024-04-24T00:46:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14091' '!=' 'object_number=63, tag_tid=11, entry_tid=13919' 2024-04-24T00:46:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14091' = 'object_number=63, tag_tid=11, entry_tid=13919' 2024-04-24T00:46:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=14091' 2024-04-24T00:46:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14091' 2024-04-24T00:46:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14091' 2024-04-24T00:46:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14091 2024-04-24T00:46:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 14091 2024-04-24T00:46:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=14288], [object_number=63, tag_tid=11, entry_tid=14287], [object_number=62, tag_tid=11, entry_tid=14286], [object_number=61, tag_tid=11, entry_tid=14285]]], state=connected] 2024-04-24T00:46:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=14288], [object_number=63, tag_tid=11, entry_tid=14287], [object_number=62, tag_tid=11, entry_tid=14286], [object_number=61, tag_tid=11, entry_tid=14285]]], state=connected] 2024-04-24T00:46:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=14288' 2024-04-24T00:46:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=14288' 2024-04-24T00:46:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=14288' '!=' 'object_number=63, tag_tid=11, entry_tid=14091' 2024-04-24T00:46:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=14288' = 'object_number=63, tag_tid=11, entry_tid=14091' 2024-04-24T00:46:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=14288' 2024-04-24T00:46:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=14288' 2024-04-24T00:46:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=14288' 2024-04-24T00:46:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14288 2024-04-24T00:46:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 14288 2024-04-24T00:46:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14443], [object_number=62, tag_tid=11, entry_tid=14442], [object_number=61, tag_tid=11, entry_tid=14441], [object_number=60, tag_tid=11, entry_tid=14440]]], state=connected] 2024-04-24T00:46:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:29.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14443], [object_number=62, tag_tid=11, entry_tid=14442], [object_number=61, tag_tid=11, entry_tid=14441], [object_number=60, tag_tid=11, entry_tid=14440]]], state=connected] 2024-04-24T00:46:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=14443' 2024-04-24T00:46:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=14443' 2024-04-24T00:46:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14443' '!=' 'object_number=60, tag_tid=11, entry_tid=14288' 2024-04-24T00:46:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14443' = 'object_number=60, tag_tid=11, entry_tid=14288' 2024-04-24T00:46:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=14443' 2024-04-24T00:46:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14443' 2024-04-24T00:46:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14443' 2024-04-24T00:46:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14443 2024-04-24T00:46:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 14443 2024-04-24T00:46:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=14732], [object_number=63, tag_tid=11, entry_tid=14731], [object_number=62, tag_tid=11, entry_tid=14730], [object_number=61, tag_tid=11, entry_tid=14729]]], state=connected] 2024-04-24T00:46:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=14732], [object_number=63, tag_tid=11, entry_tid=14731], [object_number=62, tag_tid=11, entry_tid=14730], [object_number=61, tag_tid=11, entry_tid=14729]]], state=connected] 2024-04-24T00:46:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=14732' 2024-04-24T00:46:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=14732' 2024-04-24T00:46:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=14732' '!=' 'object_number=63, tag_tid=11, entry_tid=14443' 2024-04-24T00:46:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=14732' = 'object_number=63, tag_tid=11, entry_tid=14443' 2024-04-24T00:46:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=14732' 2024-04-24T00:46:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=14732' 2024-04-24T00:46:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=14732' 2024-04-24T00:46:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14732 2024-04-24T00:46:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 14732 2024-04-24T00:46:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=14889], [object_number=60, tag_tid=11, entry_tid=14888], [object_number=63, tag_tid=11, entry_tid=14887], [object_number=62, tag_tid=11, entry_tid=14886]]], state=connected] 2024-04-24T00:46:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=14889], [object_number=60, tag_tid=11, entry_tid=14888], [object_number=63, tag_tid=11, entry_tid=14887], [object_number=62, tag_tid=11, entry_tid=14886]]], state=connected] 2024-04-24T00:46:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=14889' 2024-04-24T00:46:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=14889' 2024-04-24T00:46:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=14889' '!=' 'object_number=60, tag_tid=11, entry_tid=14732' 2024-04-24T00:46:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=14889' = 'object_number=60, tag_tid=11, entry_tid=14732' 2024-04-24T00:46:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=14889' 2024-04-24T00:46:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=14889' 2024-04-24T00:46:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=14889' 2024-04-24T00:46:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14889 2024-04-24T00:46:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 14889 2024-04-24T00:46:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=15068], [object_number=63, tag_tid=11, entry_tid=15067], [object_number=62, tag_tid=11, entry_tid=15066], [object_number=61, tag_tid=11, entry_tid=15065]]], state=connected] 2024-04-24T00:46:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=15068], [object_number=63, tag_tid=11, entry_tid=15067], [object_number=62, tag_tid=11, entry_tid=15066], [object_number=61, tag_tid=11, entry_tid=15065]]], state=connected] 2024-04-24T00:46:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=15068' 2024-04-24T00:46:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=15068' 2024-04-24T00:46:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=15068' '!=' 'object_number=61, tag_tid=11, entry_tid=14889' 2024-04-24T00:46:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=15068' = 'object_number=61, tag_tid=11, entry_tid=14889' 2024-04-24T00:46:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=15068' 2024-04-24T00:46:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=15068' 2024-04-24T00:46:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=15068' 2024-04-24T00:46:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15068 2024-04-24T00:46:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 15068 2024-04-24T00:46:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:32.789 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T00:46:32.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T00:46:32.790 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-24T00:46:32.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T00:46:32.788+0000 7fe76bb1a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 48186) UID: 1000 2024-04-24T00:46:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=15241], [object_number=60, tag_tid=11, entry_tid=15240], [object_number=63, tag_tid=11, entry_tid=15239], [object_number=62, tag_tid=11, entry_tid=15238]]], state=connected] 2024-04-24T00:46:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=15241], [object_number=60, tag_tid=11, entry_tid=15240], [object_number=63, tag_tid=11, entry_tid=15239], [object_number=62, tag_tid=11, entry_tid=15238]]], state=connected] 2024-04-24T00:46:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=15241' 2024-04-24T00:46:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=15241' 2024-04-24T00:46:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=15241' '!=' 'object_number=60, tag_tid=11, entry_tid=15068' 2024-04-24T00:46:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=15241' = 'object_number=60, tag_tid=11, entry_tid=15068' 2024-04-24T00:46:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=15241' 2024-04-24T00:46:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=15241' 2024-04-24T00:46:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=15241' 2024-04-24T00:46:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15241 2024-04-24T00:46:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 15241 2024-04-24T00:46:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=15422], [object_number=61, tag_tid=11, entry_tid=15421], [object_number=60, tag_tid=11, entry_tid=15420], [object_number=63, tag_tid=11, entry_tid=15419]]], state=connected] 2024-04-24T00:46:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=15422], [object_number=61, tag_tid=11, entry_tid=15421], [object_number=60, tag_tid=11, entry_tid=15420], [object_number=63, tag_tid=11, entry_tid=15419]]], state=connected] 2024-04-24T00:46:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=15422' 2024-04-24T00:46:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=15422' 2024-04-24T00:46:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=15422' '!=' 'object_number=61, tag_tid=11, entry_tid=15241' 2024-04-24T00:46:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=15422' = 'object_number=61, tag_tid=11, entry_tid=15241' 2024-04-24T00:46:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=15422' 2024-04-24T00:46:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=15422' 2024-04-24T00:46:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=15422' 2024-04-24T00:46:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15422 2024-04-24T00:46:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 15422 2024-04-24T00:46:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:46:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=15525], [object_number=60, tag_tid=11, entry_tid=15524], [object_number=63, tag_tid=11, entry_tid=15523], [object_number=62, tag_tid=11, entry_tid=15522]]], state=connected] 2024-04-24T00:46:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=15525], [object_number=60, tag_tid=11, entry_tid=15524], [object_number=63, tag_tid=11, entry_tid=15523], [object_number=62, tag_tid=11, entry_tid=15522]]], state=connected] 2024-04-24T00:46:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=15525' 2024-04-24T00:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=15525' 2024-04-24T00:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=15525' '!=' 'object_number=62, tag_tid=11, entry_tid=15422' 2024-04-24T00:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=15525' = 'object_number=62, tag_tid=11, entry_tid=15422' 2024-04-24T00:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=15525' 2024-04-24T00:46:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=15525' 2024-04-24T00:46:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=15525' 2024-04-24T00:46:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15525 2024-04-24T00:46:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 15525 2024-04-24T00:46:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:46:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=15711], [object_number=62, tag_tid=11, entry_tid=15710], [object_number=61, tag_tid=11, entry_tid=15709], [object_number=60, tag_tid=11, entry_tid=15708]]], state=connected] 2024-04-24T00:46:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=15711], [object_number=62, tag_tid=11, entry_tid=15710], [object_number=61, tag_tid=11, entry_tid=15709], [object_number=60, tag_tid=11, entry_tid=15708]]], state=connected] 2024-04-24T00:46:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=15711' 2024-04-24T00:46:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=15711' 2024-04-24T00:46:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=15711' '!=' 'object_number=61, tag_tid=11, entry_tid=15525' 2024-04-24T00:46:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=15711' = 'object_number=61, tag_tid=11, entry_tid=15525' 2024-04-24T00:46:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=15711' 2024-04-24T00:46:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=15711' 2024-04-24T00:46:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=15711' 2024-04-24T00:46:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15711 2024-04-24T00:46:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 15711 2024-04-24T00:46:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=15885], [object_number=60, tag_tid=11, entry_tid=15884], [object_number=63, tag_tid=11, entry_tid=15883], [object_number=62, tag_tid=11, entry_tid=15882]]], state=connected] 2024-04-24T00:46:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=15885], [object_number=60, tag_tid=11, entry_tid=15884], [object_number=63, tag_tid=11, entry_tid=15883], [object_number=62, tag_tid=11, entry_tid=15882]]], state=connected] 2024-04-24T00:46:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=15885' 2024-04-24T00:46:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=15885' 2024-04-24T00:46:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=15885' '!=' 'object_number=63, tag_tid=11, entry_tid=15711' 2024-04-24T00:46:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=15885' = 'object_number=63, tag_tid=11, entry_tid=15711' 2024-04-24T00:46:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=15885' 2024-04-24T00:46:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=15885' 2024-04-24T00:46:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=15885' 2024-04-24T00:46:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15885 2024-04-24T00:46:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 15885 2024-04-24T00:46:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:36.791 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T00:46:36.791 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:46:36.791 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T00:46:36.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T00:46:36.791 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T00:46:36.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T00:46:36.794 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-04-24T00:46:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=16046], [object_number=61, tag_tid=11, entry_tid=16045], [object_number=60, tag_tid=11, entry_tid=16044], [object_number=63, tag_tid=11, entry_tid=16043]]], state=connected] 2024-04-24T00:46:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=16046], [object_number=61, tag_tid=11, entry_tid=16045], [object_number=60, tag_tid=11, entry_tid=16044], [object_number=63, tag_tid=11, entry_tid=16043]]], state=connected] 2024-04-24T00:46:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=16046' 2024-04-24T00:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=16046' 2024-04-24T00:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=16046' '!=' 'object_number=61, tag_tid=11, entry_tid=15885' 2024-04-24T00:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=16046' = 'object_number=61, tag_tid=11, entry_tid=15885' 2024-04-24T00:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=16046' 2024-04-24T00:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=16046' 2024-04-24T00:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=16046' 2024-04-24T00:46:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16046 2024-04-24T00:46:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 16046 2024-04-24T00:46:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=16256], [object_number=63, tag_tid=11, entry_tid=16255], [object_number=62, tag_tid=11, entry_tid=16254], [object_number=61, tag_tid=11, entry_tid=16253]]], state=connected] 2024-04-24T00:46:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=16256], [object_number=63, tag_tid=11, entry_tid=16255], [object_number=62, tag_tid=11, entry_tid=16254], [object_number=61, tag_tid=11, entry_tid=16253]]], state=connected] 2024-04-24T00:46:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=16256' 2024-04-24T00:46:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=16256' 2024-04-24T00:46:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=16256' '!=' 'object_number=62, tag_tid=11, entry_tid=16046' 2024-04-24T00:46:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=16256' = 'object_number=62, tag_tid=11, entry_tid=16046' 2024-04-24T00:46:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=16256' 2024-04-24T00:46:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=16256' 2024-04-24T00:46:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=16256' 2024-04-24T00:46:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16256 2024-04-24T00:46:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 16256 2024-04-24T00:46:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=16449], [object_number=60, tag_tid=11, entry_tid=16448], [object_number=63, tag_tid=11, entry_tid=16447], [object_number=62, tag_tid=11, entry_tid=16446]]], state=connected] 2024-04-24T00:46:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=16449], [object_number=60, tag_tid=11, entry_tid=16448], [object_number=63, tag_tid=11, entry_tid=16447], [object_number=62, tag_tid=11, entry_tid=16446]]], state=connected] 2024-04-24T00:46:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=16449' 2024-04-24T00:46:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=16449' 2024-04-24T00:46:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=16449' '!=' 'object_number=60, tag_tid=11, entry_tid=16256' 2024-04-24T00:46:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=16449' = 'object_number=60, tag_tid=11, entry_tid=16256' 2024-04-24T00:46:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=16449' 2024-04-24T00:46:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=16449' 2024-04-24T00:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=16449' 2024-04-24T00:46:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16449 2024-04-24T00:46:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 16449 2024-04-24T00:46:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=16626], [object_number=61, tag_tid=11, entry_tid=16625], [object_number=60, tag_tid=11, entry_tid=16624], [object_number=63, tag_tid=11, entry_tid=16623]]], state=connected] 2024-04-24T00:46:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=16626], [object_number=61, tag_tid=11, entry_tid=16625], [object_number=60, tag_tid=11, entry_tid=16624], [object_number=63, tag_tid=11, entry_tid=16623]]], state=connected] 2024-04-24T00:46:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=16626' 2024-04-24T00:46:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=16626' 2024-04-24T00:46:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=16626' '!=' 'object_number=61, tag_tid=11, entry_tid=16449' 2024-04-24T00:46:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=16626' = 'object_number=61, tag_tid=11, entry_tid=16449' 2024-04-24T00:46:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=16626' 2024-04-24T00:46:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=16626' 2024-04-24T00:46:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=16626' 2024-04-24T00:46:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16626 2024-04-24T00:46:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 16626 2024-04-24T00:46:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=16794], [object_number=61, tag_tid=11, entry_tid=16793], [object_number=60, tag_tid=11, entry_tid=16792], [object_number=63, tag_tid=11, entry_tid=16791]]], state=connected] 2024-04-24T00:46:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=16794], [object_number=61, tag_tid=11, entry_tid=16793], [object_number=60, tag_tid=11, entry_tid=16792], [object_number=63, tag_tid=11, entry_tid=16791]]], state=connected] 2024-04-24T00:46:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=16794' 2024-04-24T00:46:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=16794' 2024-04-24T00:46:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=16794' '!=' 'object_number=62, tag_tid=11, entry_tid=16626' 2024-04-24T00:46:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=16794' = 'object_number=62, tag_tid=11, entry_tid=16626' 2024-04-24T00:46:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=16794' 2024-04-24T00:46:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=16794' 2024-04-24T00:46:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=16794' 2024-04-24T00:46:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16794 2024-04-24T00:46:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 16794 2024-04-24T00:46:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=16951], [object_number=62, tag_tid=11, entry_tid=16950], [object_number=61, tag_tid=11, entry_tid=16949], [object_number=60, tag_tid=11, entry_tid=16948]]], state=connected] 2024-04-24T00:46:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=16951], [object_number=62, tag_tid=11, entry_tid=16950], [object_number=61, tag_tid=11, entry_tid=16949], [object_number=60, tag_tid=11, entry_tid=16948]]], state=connected] 2024-04-24T00:46:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=16951' 2024-04-24T00:46:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=16951' 2024-04-24T00:46:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=16951' '!=' 'object_number=62, tag_tid=11, entry_tid=16794' 2024-04-24T00:46:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=16951' = 'object_number=62, tag_tid=11, entry_tid=16794' 2024-04-24T00:46:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=16951' 2024-04-24T00:46:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=16951' 2024-04-24T00:46:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=16951' 2024-04-24T00:46:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16951 2024-04-24T00:46:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 16951 2024-04-24T00:46:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=17109], [object_number=60, tag_tid=11, entry_tid=17108], [object_number=63, tag_tid=11, entry_tid=17107], [object_number=62, tag_tid=11, entry_tid=17106]]], state=connected] 2024-04-24T00:46:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 15 2024-04-24T00:46:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=17109], [object_number=60, tag_tid=11, entry_tid=17108], [object_number=63, tag_tid=11, entry_tid=17107], [object_number=62, tag_tid=11, entry_tid=17106]]], state=connected] 2024-04-24T00:46:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=17109' 2024-04-24T00:46:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=17109' 2024-04-24T00:46:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=17109' '!=' 'object_number=63, tag_tid=11, entry_tid=16951' 2024-04-24T00:46:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=17109' = 'object_number=63, tag_tid=11, entry_tid=16951' 2024-04-24T00:46:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=17109' 2024-04-24T00:46:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=17109' 2024-04-24T00:46:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=17109' 2024-04-24T00:46:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17109 2024-04-24T00:46:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 17109 2024-04-24T00:46:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=17270], [object_number=65, tag_tid=11, entry_tid=17269], [object_number=64, tag_tid=11, entry_tid=17268], [object_number=67, tag_tid=11, entry_tid=17267]]], state=connected] 2024-04-24T00:46:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=17270], [object_number=65, tag_tid=11, entry_tid=17269], [object_number=64, tag_tid=11, entry_tid=17268], [object_number=67, tag_tid=11, entry_tid=17267]]], state=connected] 2024-04-24T00:46:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=66, tag_tid=11, entry_tid=17270' 2024-04-24T00:46:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=66, tag_tid=11, entry_tid=17270' 2024-04-24T00:46:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=17270' '!=' 'object_number=61, tag_tid=11, entry_tid=17109' 2024-04-24T00:46:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=17270' = 'object_number=61, tag_tid=11, entry_tid=17109' 2024-04-24T00:46:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=66, tag_tid=11, entry_tid=17270' 2024-04-24T00:46:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=17270' 2024-04-24T00:46:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=17270' 2024-04-24T00:46:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17270 2024-04-24T00:46:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 17270 2024-04-24T00:46:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=65, tag_tid=11, entry_tid=17441], [object_number=64, tag_tid=11, entry_tid=17440], [object_number=67, tag_tid=11, entry_tid=17439], [object_number=66, tag_tid=11, entry_tid=17438]]], state=connected] 2024-04-24T00:46:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=65, tag_tid=11, entry_tid=17441], [object_number=64, tag_tid=11, entry_tid=17440], [object_number=67, tag_tid=11, entry_tid=17439], [object_number=66, tag_tid=11, entry_tid=17438]]], state=connected] 2024-04-24T00:46:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=65, tag_tid=11, entry_tid=17441' 2024-04-24T00:46:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=65, tag_tid=11, entry_tid=17441' 2024-04-24T00:46:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=65, tag_tid=11, entry_tid=17441' '!=' 'object_number=66, tag_tid=11, entry_tid=17270' 2024-04-24T00:46:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=65, tag_tid=11, entry_tid=17441' = 'object_number=66, tag_tid=11, entry_tid=17270' 2024-04-24T00:46:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=65, tag_tid=11, entry_tid=17441' 2024-04-24T00:46:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=11, entry_tid=17441' 2024-04-24T00:46:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=11, entry_tid=17441' 2024-04-24T00:46:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17441 2024-04-24T00:46:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 17441 2024-04-24T00:46:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=17615], [object_number=66, tag_tid=11, entry_tid=17614], [object_number=65, tag_tid=11, entry_tid=17613], [object_number=64, tag_tid=11, entry_tid=17612]]], state=connected] 2024-04-24T00:46:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=17615], [object_number=66, tag_tid=11, entry_tid=17614], [object_number=65, tag_tid=11, entry_tid=17613], [object_number=64, tag_tid=11, entry_tid=17612]]], state=connected] 2024-04-24T00:46:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=17615' 2024-04-24T00:46:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=17615' 2024-04-24T00:46:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=17615' '!=' 'object_number=65, tag_tid=11, entry_tid=17441' 2024-04-24T00:46:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=17615' = 'object_number=65, tag_tid=11, entry_tid=17441' 2024-04-24T00:46:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=17615' 2024-04-24T00:46:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=17615' 2024-04-24T00:46:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=17615' 2024-04-24T00:46:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17615 2024-04-24T00:46:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 17615 2024-04-24T00:46:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:45.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:45.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=17810], [object_number=65, tag_tid=11, entry_tid=17809], [object_number=64, tag_tid=11, entry_tid=17808], [object_number=67, tag_tid=11, entry_tid=17807]]], state=connected] 2024-04-24T00:46:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=17810], [object_number=65, tag_tid=11, entry_tid=17809], [object_number=64, tag_tid=11, entry_tid=17808], [object_number=67, tag_tid=11, entry_tid=17807]]], state=connected] 2024-04-24T00:46:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=66, tag_tid=11, entry_tid=17810' 2024-04-24T00:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=66, tag_tid=11, entry_tid=17810' 2024-04-24T00:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=17810' '!=' 'object_number=67, tag_tid=11, entry_tid=17615' 2024-04-24T00:46:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=17810' = 'object_number=67, tag_tid=11, entry_tid=17615' 2024-04-24T00:46:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=66, tag_tid=11, entry_tid=17810' 2024-04-24T00:46:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=17810' 2024-04-24T00:46:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=17810' 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17810 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 17810 2024-04-24T00:46:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=17848], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846], [object_number=65, tag_tid=11, entry_tid=17845]]], state=connected] 2024-04-24T00:46:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=17848], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846], [object_number=65, tag_tid=11, entry_tid=17845]]], state=connected] 2024-04-24T00:46:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=17848' 2024-04-24T00:46:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=17848' 2024-04-24T00:46:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=17848' '!=' 'object_number=66, tag_tid=11, entry_tid=17810' 2024-04-24T00:46:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:46:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=17848' = 'object_number=66, tag_tid=11, entry_tid=17810' 2024-04-24T00:46:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=17848' 2024-04-24T00:46:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=12 2024-04-24T00:46:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=17848' 2024-04-24T00:46:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:46:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=11 2024-04-24T00:46:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:46:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=17848' 2024-04-24T00:46:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:46:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:46:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17848 2024-04-24T00:46:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 12 = 11 -a 1 -le 17848 2024-04-24T00:46:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:46:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:46:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:46:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:46:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:46:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:46:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:46:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:46:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:46:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:46:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:46:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:46:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:46:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:46:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:46:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:46:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:46:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:46:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:46:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:46:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:46:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:46:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:46:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:46:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:46:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:46:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:46:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:46:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:46:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:46:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:46:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:46:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:46:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:46:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:46:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:46:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:46:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:46:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:46:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:46:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:46:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:46:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:46:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:46:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:46:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:46:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:46:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:46:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:46:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:46:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:46:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:46:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:46:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:46:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:46:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:46:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:46:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:46:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:46:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:46:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:46:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:46:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:46:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:46:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:46:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:46:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:46:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:46:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:46:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:46:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 16 2024-04-24T00:46:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 16 2024-04-24T00:46:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:46:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=17847], [object_number=66, tag_tid=11, entry_tid=17846]]], state=connected] 2024-04-24T00:46:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:46:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=65, tag_tid=12, entry_tid=1' 2024-04-24T00:46:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-04-24T00:46:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:46:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4 2024-04-24T00:46:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:46:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:46:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:46:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-04-24T00:46:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:46:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:46:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 7 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:44:31 2024 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:46:00 2024 2024-04-24T00:46:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:46:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:46:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:46:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:46:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:46:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap3-clean ']' 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3-clean 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:46:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:46:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:50.229 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:46:50.228+0000 7efd534d1640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:46:50.230 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:46:50.228+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:46:50.232 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:46:50.228+0000 7efd514cd640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:46:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:46:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap4 2024-04-24T00:46:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4 2024-04-24T00:46:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:46:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:46:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:46:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:46:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:52.352 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:46:52.348+0000 7efd524cf640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T00:46:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:46:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:46:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:46:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap4-clean 2024-04-24T00:46:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-04-24T00:46:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:46:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4-clean 2024-04-24T00:46:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:46:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:46:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:46:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:46:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:46:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:46:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:46:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T00:46:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:46:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:46:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 92541952 bytes 2024-04-24T00:46:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 106151936 bytes 2024-04-24T00:46:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 135483392 bytes 2024-04-24T00:47:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 185423872 bytes 2024-04-24T00:47:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 233705472 bytes 2024-04-24T00:47:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 268318720 bytes 2024-04-24T00:47:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 299321344 bytes 2024-04-24T00:47:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 321312768 bytes 2024-04-24T00:47:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 344824832 bytes 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:47:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:47:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:47:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:47:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:47:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:47:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:47:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:47:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:47:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":1876311.35,"entries_behind_primary":0,"entries_per_second":119.15,"non_primary_position":{"entry_tid":1,"object_number":65,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":65,"tag_tid":12}} 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.3 on smithi134 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:46:56 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:47:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:47:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:46:57' 2024-04-24T00:47:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:47:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:47:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:47:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":1876311.35,"entries_behind_primary":0,"entries_per_second":119.15,"non_primary_position":{"entry_tid":1,"object_number":65,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":65,"tag_tid":12}} 2024-04-24T00:47:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:47:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:47:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:47:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:47:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:47:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.3 on smithi134 2024-04-24T00:47:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:47:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:47:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:46:56' 2024-04-24T00:47:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:46:56' 2024-04-24T00:47:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:47:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:46:56' 2024-04-24T00:47:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:47:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:47:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:47:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142194274mirror.3smithi13419.0.0-2424-gb98aab6ctrueOK42594312mirror.2smithi13419.0.0-2424-gb98aab6cfalseOK44094412mirror.1smithi13419.0.0-2424-gb98aab6cfalseOK44234426mirror.0smithi13419.0.0-2424-gb98aab6cfalseOKtestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":1876311.35,"entries_behind_primary":0,"entries_per_second":119.15,"non_primary_position":{"entry_tid":1,"object_number":65,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":65,"tag_tid":12}}42194274mirror.3smithi1342024-04-24 00:46:56cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:46:57' 2024-04-24T00:47:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:47:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:47:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:46:56' 2024-04-24T00:47:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:47:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:47:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:47:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:47:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:47:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5 2024-04-24T00:47:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap5 2024-04-24T00:47:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:47:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5 2024-04-24T00:47:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:47:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:47:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:47:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:47:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:47:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:47:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:47:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:47:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:47:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:47:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:47:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:47:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:47:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:47:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:47:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:47:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:47:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:47:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:47:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:47:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:47:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:47:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:47:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:47:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:47:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:47:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:47:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:47:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:47:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:47:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:47:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:47:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:47:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:47:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:47:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:47:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:47:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:47:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:47:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:47:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:47:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:47:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:47:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:47:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:47:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:47:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:47:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:47:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:47:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:47:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:47:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:47:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:47:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:47:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:47:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:47:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:47:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:47:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:47:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:47:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:47:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:47:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=71, tag_tid=14, entry_tid=6979], [object_number=70, tag_tid=14, entry_tid=6978], [object_number=69, tag_tid=14, entry_tid=6977], [object_number=68, tag_tid=14, entry_tid=6976]]], state=connected] 2024-04-24T00:47:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=71, tag_tid=14, entry_tid=6979], [object_number=70, tag_tid=14, entry_tid=6978], [object_number=69, tag_tid=14, entry_tid=6977], [object_number=68, tag_tid=14, entry_tid=6976]]], state=connected] 2024-04-24T00:47:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=71, tag_tid=14, entry_tid=6979' 2024-04-24T00:47:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=71, tag_tid=14, entry_tid=6979' 2024-04-24T00:47:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=71, tag_tid=14, entry_tid=6979' '!=' '' 2024-04-24T00:47:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=71, tag_tid=14, entry_tid=6979' = '' 2024-04-24T00:47:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=71, tag_tid=14, entry_tid=6979' 2024-04-24T00:47:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=71, tag_tid=14, entry_tid=6979' 2024-04-24T00:47:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=71, tag_tid=14, entry_tid=6979' 2024-04-24T00:47:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6979 2024-04-24T00:47:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 6979 2024-04-24T00:47:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=68, tag_tid=14, entry_tid=7156], [object_number=71, tag_tid=14, entry_tid=7155], [object_number=70, tag_tid=14, entry_tid=7154], [object_number=69, tag_tid=14, entry_tid=7153]]], state=connected] 2024-04-24T00:47:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=68, tag_tid=14, entry_tid=7156], [object_number=71, tag_tid=14, entry_tid=7155], [object_number=70, tag_tid=14, entry_tid=7154], [object_number=69, tag_tid=14, entry_tid=7153]]], state=connected] 2024-04-24T00:47:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=68, tag_tid=14, entry_tid=7156' 2024-04-24T00:47:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=68, tag_tid=14, entry_tid=7156' 2024-04-24T00:47:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=68, tag_tid=14, entry_tid=7156' '!=' 'object_number=71, tag_tid=14, entry_tid=6979' 2024-04-24T00:47:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=68, tag_tid=14, entry_tid=7156' = 'object_number=71, tag_tid=14, entry_tid=6979' 2024-04-24T00:47:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=68, tag_tid=14, entry_tid=7156' 2024-04-24T00:47:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=68, tag_tid=14, entry_tid=7156' 2024-04-24T00:47:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=68, tag_tid=14, entry_tid=7156' 2024-04-24T00:47:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7156 2024-04-24T00:47:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 7156 2024-04-24T00:47:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:42.900 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T00:47:42.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T00:47:42.900 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-24T00:47:42.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T00:47:42.899+0000 7f4acd264640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 61737) UID: 1000 2024-04-24T00:47:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=71, tag_tid=14, entry_tid=7343], [object_number=70, tag_tid=14, entry_tid=7342], [object_number=69, tag_tid=14, entry_tid=7341], [object_number=68, tag_tid=14, entry_tid=7340]]], state=connected] 2024-04-24T00:47:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=71, tag_tid=14, entry_tid=7343], [object_number=70, tag_tid=14, entry_tid=7342], [object_number=69, tag_tid=14, entry_tid=7341], [object_number=68, tag_tid=14, entry_tid=7340]]], state=connected] 2024-04-24T00:47:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=71, tag_tid=14, entry_tid=7343' 2024-04-24T00:47:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=71, tag_tid=14, entry_tid=7343' 2024-04-24T00:47:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=71, tag_tid=14, entry_tid=7343' '!=' 'object_number=68, tag_tid=14, entry_tid=7156' 2024-04-24T00:47:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=71, tag_tid=14, entry_tid=7343' = 'object_number=68, tag_tid=14, entry_tid=7156' 2024-04-24T00:47:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=71, tag_tid=14, entry_tid=7343' 2024-04-24T00:47:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=71, tag_tid=14, entry_tid=7343' 2024-04-24T00:47:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=71, tag_tid=14, entry_tid=7343' 2024-04-24T00:47:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7343 2024-04-24T00:47:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 7343 2024-04-24T00:47:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=68, tag_tid=14, entry_tid=7548], [object_number=71, tag_tid=14, entry_tid=7547], [object_number=70, tag_tid=14, entry_tid=7546], [object_number=69, tag_tid=14, entry_tid=7545]]], state=connected] 2024-04-24T00:47:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=68, tag_tid=14, entry_tid=7548], [object_number=71, tag_tid=14, entry_tid=7547], [object_number=70, tag_tid=14, entry_tid=7546], [object_number=69, tag_tid=14, entry_tid=7545]]], state=connected] 2024-04-24T00:47:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=68, tag_tid=14, entry_tid=7548' 2024-04-24T00:47:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=68, tag_tid=14, entry_tid=7548' 2024-04-24T00:47:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=68, tag_tid=14, entry_tid=7548' '!=' 'object_number=71, tag_tid=14, entry_tid=7343' 2024-04-24T00:47:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=68, tag_tid=14, entry_tid=7548' = 'object_number=71, tag_tid=14, entry_tid=7343' 2024-04-24T00:47:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=68, tag_tid=14, entry_tid=7548' 2024-04-24T00:47:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=68, tag_tid=14, entry_tid=7548' 2024-04-24T00:47:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=68, tag_tid=14, entry_tid=7548' 2024-04-24T00:47:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7548 2024-04-24T00:47:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 7548 2024-04-24T00:47:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:44.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:44.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:44.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:44.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=68, tag_tid=14, entry_tid=7704], [object_number=71, tag_tid=14, entry_tid=7703], [object_number=70, tag_tid=14, entry_tid=7702], [object_number=69, tag_tid=14, entry_tid=7701]]], state=connected] 2024-04-24T00:47:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 17 2024-04-24T00:47:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:45.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=68, tag_tid=14, entry_tid=7704], [object_number=71, tag_tid=14, entry_tid=7703], [object_number=70, tag_tid=14, entry_tid=7702], [object_number=69, tag_tid=14, entry_tid=7701]]], state=connected] 2024-04-24T00:47:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=68, tag_tid=14, entry_tid=7704' 2024-04-24T00:47:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=68, tag_tid=14, entry_tid=7704' 2024-04-24T00:47:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=68, tag_tid=14, entry_tid=7704' '!=' 'object_number=68, tag_tid=14, entry_tid=7548' 2024-04-24T00:47:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=68, tag_tid=14, entry_tid=7704' = 'object_number=68, tag_tid=14, entry_tid=7548' 2024-04-24T00:47:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=68, tag_tid=14, entry_tid=7704' 2024-04-24T00:47:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=68, tag_tid=14, entry_tid=7704' 2024-04-24T00:47:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=68, tag_tid=14, entry_tid=7704' 2024-04-24T00:47:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7704 2024-04-24T00:47:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 7704 2024-04-24T00:47:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=7853], [object_number=72, tag_tid=14, entry_tid=7852], [object_number=75, tag_tid=14, entry_tid=7851], [object_number=74, tag_tid=14, entry_tid=7850]]], state=connected] 2024-04-24T00:47:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=7853], [object_number=72, tag_tid=14, entry_tid=7852], [object_number=75, tag_tid=14, entry_tid=7851], [object_number=74, tag_tid=14, entry_tid=7850]]], state=connected] 2024-04-24T00:47:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=73, tag_tid=14, entry_tid=7853' 2024-04-24T00:47:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=14, entry_tid=7853' 2024-04-24T00:47:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=7853' '!=' 'object_number=68, tag_tid=14, entry_tid=7704' 2024-04-24T00:47:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=7853' = 'object_number=68, tag_tid=14, entry_tid=7704' 2024-04-24T00:47:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=73, tag_tid=14, entry_tid=7853' 2024-04-24T00:47:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=7853' 2024-04-24T00:47:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=7853' 2024-04-24T00:47:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7853 2024-04-24T00:47:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 7853 2024-04-24T00:47:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=74, tag_tid=14, entry_tid=8090], [object_number=73, tag_tid=14, entry_tid=8089], [object_number=72, tag_tid=14, entry_tid=8088], [object_number=75, tag_tid=14, entry_tid=8087]]], state=connected] 2024-04-24T00:47:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=74, tag_tid=14, entry_tid=8090], [object_number=73, tag_tid=14, entry_tid=8089], [object_number=72, tag_tid=14, entry_tid=8088], [object_number=75, tag_tid=14, entry_tid=8087]]], state=connected] 2024-04-24T00:47:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=74, tag_tid=14, entry_tid=8090' 2024-04-24T00:47:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=74, tag_tid=14, entry_tid=8090' 2024-04-24T00:47:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=74, tag_tid=14, entry_tid=8090' '!=' 'object_number=73, tag_tid=14, entry_tid=7853' 2024-04-24T00:47:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=74, tag_tid=14, entry_tid=8090' = 'object_number=73, tag_tid=14, entry_tid=7853' 2024-04-24T00:47:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=74, tag_tid=14, entry_tid=8090' 2024-04-24T00:47:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=74, tag_tid=14, entry_tid=8090' 2024-04-24T00:47:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=74, tag_tid=14, entry_tid=8090' 2024-04-24T00:47:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8090 2024-04-24T00:47:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 8090 2024-04-24T00:47:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=8268], [object_number=75, tag_tid=14, entry_tid=8267], [object_number=74, tag_tid=14, entry_tid=8266], [object_number=73, tag_tid=14, entry_tid=8265]]], state=connected] 2024-04-24T00:47:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=8268], [object_number=75, tag_tid=14, entry_tid=8267], [object_number=74, tag_tid=14, entry_tid=8266], [object_number=73, tag_tid=14, entry_tid=8265]]], state=connected] 2024-04-24T00:47:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=72, tag_tid=14, entry_tid=8268' 2024-04-24T00:47:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=72, tag_tid=14, entry_tid=8268' 2024-04-24T00:47:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=8268' '!=' 'object_number=74, tag_tid=14, entry_tid=8090' 2024-04-24T00:47:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=8268' = 'object_number=74, tag_tid=14, entry_tid=8090' 2024-04-24T00:47:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=72, tag_tid=14, entry_tid=8268' 2024-04-24T00:47:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=8268' 2024-04-24T00:47:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=8268' 2024-04-24T00:47:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8268 2024-04-24T00:47:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 8268 2024-04-24T00:47:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=74, tag_tid=14, entry_tid=8390], [object_number=73, tag_tid=14, entry_tid=8389], [object_number=72, tag_tid=14, entry_tid=8388], [object_number=75, tag_tid=14, entry_tid=8387]]], state=connected] 2024-04-24T00:47:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=74, tag_tid=14, entry_tid=8390], [object_number=73, tag_tid=14, entry_tid=8389], [object_number=72, tag_tid=14, entry_tid=8388], [object_number=75, tag_tid=14, entry_tid=8387]]], state=connected] 2024-04-24T00:47:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=74, tag_tid=14, entry_tid=8390' 2024-04-24T00:47:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=74, tag_tid=14, entry_tid=8390' 2024-04-24T00:47:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=74, tag_tid=14, entry_tid=8390' '!=' 'object_number=72, tag_tid=14, entry_tid=8268' 2024-04-24T00:47:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=74, tag_tid=14, entry_tid=8390' = 'object_number=72, tag_tid=14, entry_tid=8268' 2024-04-24T00:47:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=74, tag_tid=14, entry_tid=8390' 2024-04-24T00:47:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=74, tag_tid=14, entry_tid=8390' 2024-04-24T00:47:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=74, tag_tid=14, entry_tid=8390' 2024-04-24T00:47:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8390 2024-04-24T00:47:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 8390 2024-04-24T00:47:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:49.901 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T00:47:49.901 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:47:49.902 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T00:47:49.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T00:47:49.902 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T00:47:49.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T00:47:49.905 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-04-24T00:47:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=8559], [object_number=74, tag_tid=14, entry_tid=8558], [object_number=73, tag_tid=14, entry_tid=8557], [object_number=72, tag_tid=14, entry_tid=8556]]], state=connected] 2024-04-24T00:47:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=8559], [object_number=74, tag_tid=14, entry_tid=8558], [object_number=73, tag_tid=14, entry_tid=8557], [object_number=72, tag_tid=14, entry_tid=8556]]], state=connected] 2024-04-24T00:47:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=75, tag_tid=14, entry_tid=8559' 2024-04-24T00:47:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=75, tag_tid=14, entry_tid=8559' 2024-04-24T00:47:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=8559' '!=' 'object_number=74, tag_tid=14, entry_tid=8390' 2024-04-24T00:47:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=8559' = 'object_number=74, tag_tid=14, entry_tid=8390' 2024-04-24T00:47:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=75, tag_tid=14, entry_tid=8559' 2024-04-24T00:47:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=8559' 2024-04-24T00:47:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=8559' 2024-04-24T00:47:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8559 2024-04-24T00:47:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 8559 2024-04-24T00:47:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=8753], [object_number=72, tag_tid=14, entry_tid=8752], [object_number=75, tag_tid=14, entry_tid=8751], [object_number=74, tag_tid=14, entry_tid=8750]]], state=connected] 2024-04-24T00:47:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=8753], [object_number=72, tag_tid=14, entry_tid=8752], [object_number=75, tag_tid=14, entry_tid=8751], [object_number=74, tag_tid=14, entry_tid=8750]]], state=connected] 2024-04-24T00:47:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=73, tag_tid=14, entry_tid=8753' 2024-04-24T00:47:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=14, entry_tid=8753' 2024-04-24T00:47:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=8753' '!=' 'object_number=75, tag_tid=14, entry_tid=8559' 2024-04-24T00:47:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=8753' = 'object_number=75, tag_tid=14, entry_tid=8559' 2024-04-24T00:47:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=73, tag_tid=14, entry_tid=8753' 2024-04-24T00:47:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=8753' 2024-04-24T00:47:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=8753' 2024-04-24T00:47:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8753 2024-04-24T00:47:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 8753 2024-04-24T00:47:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=74, tag_tid=14, entry_tid=8946], [object_number=73, tag_tid=14, entry_tid=8945], [object_number=72, tag_tid=14, entry_tid=8944], [object_number=75, tag_tid=14, entry_tid=8943]]], state=connected] 2024-04-24T00:47:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=74, tag_tid=14, entry_tid=8946], [object_number=73, tag_tid=14, entry_tid=8945], [object_number=72, tag_tid=14, entry_tid=8944], [object_number=75, tag_tid=14, entry_tid=8943]]], state=connected] 2024-04-24T00:47:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=74, tag_tid=14, entry_tid=8946' 2024-04-24T00:47:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=74, tag_tid=14, entry_tid=8946' 2024-04-24T00:47:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=74, tag_tid=14, entry_tid=8946' '!=' 'object_number=73, tag_tid=14, entry_tid=8753' 2024-04-24T00:47:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=74, tag_tid=14, entry_tid=8946' = 'object_number=73, tag_tid=14, entry_tid=8753' 2024-04-24T00:47:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=74, tag_tid=14, entry_tid=8946' 2024-04-24T00:47:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=74, tag_tid=14, entry_tid=8946' 2024-04-24T00:47:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=74, tag_tid=14, entry_tid=8946' 2024-04-24T00:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8946 2024-04-24T00:47:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 8946 2024-04-24T00:47:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=9123], [object_number=74, tag_tid=14, entry_tid=9122], [object_number=73, tag_tid=14, entry_tid=9121], [object_number=72, tag_tid=14, entry_tid=9120]]], state=connected] 2024-04-24T00:47:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=9123], [object_number=74, tag_tid=14, entry_tid=9122], [object_number=73, tag_tid=14, entry_tid=9121], [object_number=72, tag_tid=14, entry_tid=9120]]], state=connected] 2024-04-24T00:47:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=75, tag_tid=14, entry_tid=9123' 2024-04-24T00:47:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=75, tag_tid=14, entry_tid=9123' 2024-04-24T00:47:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=9123' '!=' 'object_number=74, tag_tid=14, entry_tid=8946' 2024-04-24T00:47:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=9123' = 'object_number=74, tag_tid=14, entry_tid=8946' 2024-04-24T00:47:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=75, tag_tid=14, entry_tid=9123' 2024-04-24T00:47:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=9123' 2024-04-24T00:47:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=9123' 2024-04-24T00:47:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9123 2024-04-24T00:47:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 9123 2024-04-24T00:47:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=9297], [object_number=72, tag_tid=14, entry_tid=9296], [object_number=75, tag_tid=14, entry_tid=9295], [object_number=74, tag_tid=14, entry_tid=9294]]], state=connected] 2024-04-24T00:47:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=9297], [object_number=72, tag_tid=14, entry_tid=9296], [object_number=75, tag_tid=14, entry_tid=9295], [object_number=74, tag_tid=14, entry_tid=9294]]], state=connected] 2024-04-24T00:47:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=73, tag_tid=14, entry_tid=9297' 2024-04-24T00:47:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=14, entry_tid=9297' 2024-04-24T00:47:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=9297' '!=' 'object_number=75, tag_tid=14, entry_tid=9123' 2024-04-24T00:47:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=9297' = 'object_number=75, tag_tid=14, entry_tid=9123' 2024-04-24T00:47:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=73, tag_tid=14, entry_tid=9297' 2024-04-24T00:47:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=9297' 2024-04-24T00:47:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=9297' 2024-04-24T00:47:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9297 2024-04-24T00:47:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 9297 2024-04-24T00:47:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=9456], [object_number=75, tag_tid=14, entry_tid=9455], [object_number=74, tag_tid=14, entry_tid=9454], [object_number=73, tag_tid=14, entry_tid=9453]]], state=connected] 2024-04-24T00:47:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=9456], [object_number=75, tag_tid=14, entry_tid=9455], [object_number=74, tag_tid=14, entry_tid=9454], [object_number=73, tag_tid=14, entry_tid=9453]]], state=connected] 2024-04-24T00:47:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=72, tag_tid=14, entry_tid=9456' 2024-04-24T00:47:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=72, tag_tid=14, entry_tid=9456' 2024-04-24T00:47:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=9456' '!=' 'object_number=73, tag_tid=14, entry_tid=9297' 2024-04-24T00:47:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=9456' = 'object_number=73, tag_tid=14, entry_tid=9297' 2024-04-24T00:47:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=72, tag_tid=14, entry_tid=9456' 2024-04-24T00:47:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=9456' 2024-04-24T00:47:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=9456' 2024-04-24T00:47:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9456 2024-04-24T00:47:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 9456 2024-04-24T00:47:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:55.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:55.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=9635], [object_number=74, tag_tid=14, entry_tid=9634], [object_number=73, tag_tid=14, entry_tid=9633], [object_number=72, tag_tid=14, entry_tid=9632]]], state=connected] 2024-04-24T00:47:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=9635], [object_number=74, tag_tid=14, entry_tid=9634], [object_number=73, tag_tid=14, entry_tid=9633], [object_number=72, tag_tid=14, entry_tid=9632]]], state=connected] 2024-04-24T00:47:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=75, tag_tid=14, entry_tid=9635' 2024-04-24T00:47:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=75, tag_tid=14, entry_tid=9635' 2024-04-24T00:47:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=9635' '!=' 'object_number=72, tag_tid=14, entry_tid=9456' 2024-04-24T00:47:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=9635' = 'object_number=72, tag_tid=14, entry_tid=9456' 2024-04-24T00:47:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=75, tag_tid=14, entry_tid=9635' 2024-04-24T00:47:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=9635' 2024-04-24T00:47:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=9635' 2024-04-24T00:47:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9635 2024-04-24T00:47:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 9635 2024-04-24T00:47:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=9780], [object_number=75, tag_tid=14, entry_tid=9779], [object_number=74, tag_tid=14, entry_tid=9778], [object_number=73, tag_tid=14, entry_tid=9777]]], state=connected] 2024-04-24T00:47:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=9780], [object_number=75, tag_tid=14, entry_tid=9779], [object_number=74, tag_tid=14, entry_tid=9778], [object_number=73, tag_tid=14, entry_tid=9777]]], state=connected] 2024-04-24T00:47:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=72, tag_tid=14, entry_tid=9780' 2024-04-24T00:47:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=72, tag_tid=14, entry_tid=9780' 2024-04-24T00:47:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=9780' '!=' 'object_number=75, tag_tid=14, entry_tid=9635' 2024-04-24T00:47:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=9780' = 'object_number=75, tag_tid=14, entry_tid=9635' 2024-04-24T00:47:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=72, tag_tid=14, entry_tid=9780' 2024-04-24T00:47:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=9780' 2024-04-24T00:47:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=9780' 2024-04-24T00:47:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9780 2024-04-24T00:47:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 9780 2024-04-24T00:47:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:57.381 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:47:57.383+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa099a00 2024-04-24T00:47:57.381 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:47:57.383+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa7900d0 2024-04-24T00:47:57.381 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:47:57.383+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa791520 2024-04-24T00:47:57.381 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:47:57.383+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db2616000 2024-04-24T00:47:57.382 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:47:57.383+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc0178680 2024-04-24T00:47:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=9956], [object_number=75, tag_tid=14, entry_tid=9955], [object_number=74, tag_tid=14, entry_tid=9954], [object_number=73, tag_tid=14, entry_tid=9953]]], state=connected] 2024-04-24T00:47:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=9956], [object_number=75, tag_tid=14, entry_tid=9955], [object_number=74, tag_tid=14, entry_tid=9954], [object_number=73, tag_tid=14, entry_tid=9953]]], state=connected] 2024-04-24T00:47:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=72, tag_tid=14, entry_tid=9956' 2024-04-24T00:47:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=72, tag_tid=14, entry_tid=9956' 2024-04-24T00:47:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=9956' '!=' 'object_number=72, tag_tid=14, entry_tid=9780' 2024-04-24T00:47:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=9956' = 'object_number=72, tag_tid=14, entry_tid=9780' 2024-04-24T00:47:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=72, tag_tid=14, entry_tid=9956' 2024-04-24T00:47:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=9956' 2024-04-24T00:47:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=9956' 2024-04-24T00:47:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9956 2024-04-24T00:47:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 9956 2024-04-24T00:47:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=10125], [object_number=72, tag_tid=14, entry_tid=10124], [object_number=75, tag_tid=14, entry_tid=10123], [object_number=74, tag_tid=14, entry_tid=10122]]], state=connected] 2024-04-24T00:47:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=10125], [object_number=72, tag_tid=14, entry_tid=10124], [object_number=75, tag_tid=14, entry_tid=10123], [object_number=74, tag_tid=14, entry_tid=10122]]], state=connected] 2024-04-24T00:47:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=73, tag_tid=14, entry_tid=10125' 2024-04-24T00:47:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=14, entry_tid=10125' 2024-04-24T00:47:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=10125' '!=' 'object_number=72, tag_tid=14, entry_tid=9956' 2024-04-24T00:47:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=10125' = 'object_number=72, tag_tid=14, entry_tid=9956' 2024-04-24T00:47:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=73, tag_tid=14, entry_tid=10125' 2024-04-24T00:47:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=10125' 2024-04-24T00:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=10125' 2024-04-24T00:47:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10125 2024-04-24T00:47:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 10125 2024-04-24T00:47:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=10287], [object_number=74, tag_tid=14, entry_tid=10286], [object_number=73, tag_tid=14, entry_tid=10285], [object_number=72, tag_tid=14, entry_tid=10284]]], state=connected] 2024-04-24T00:47:59.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:47:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:47:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:47:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:47:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:47:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:47:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:47:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=10287], [object_number=74, tag_tid=14, entry_tid=10286], [object_number=73, tag_tid=14, entry_tid=10285], [object_number=72, tag_tid=14, entry_tid=10284]]], state=connected] 2024-04-24T00:47:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=75, tag_tid=14, entry_tid=10287' 2024-04-24T00:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=75, tag_tid=14, entry_tid=10287' 2024-04-24T00:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=10287' '!=' 'object_number=73, tag_tid=14, entry_tid=10125' 2024-04-24T00:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=10287' = 'object_number=73, tag_tid=14, entry_tid=10125' 2024-04-24T00:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=75, tag_tid=14, entry_tid=10287' 2024-04-24T00:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:47:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=10287' 2024-04-24T00:47:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:47:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:47:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:47:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=10287' 2024-04-24T00:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:47:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10287 2024-04-24T00:47:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 10287 2024-04-24T00:47:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:47:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:47:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:47:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:47:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:47:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:47:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:47:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:47:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:47:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:47:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:47:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:47:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:47:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:47:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:47:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:47:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:47:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:47:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:47:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:47:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:47:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:47:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:47:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:47:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:47:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:47:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:47:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:47:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:47:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:47:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:47:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:47:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:47:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:47:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:47:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:47:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:47:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:47:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:47:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:47:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:47:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:47:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:47:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:47:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:47:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:47:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:47:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:47:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:47:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:47:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:47:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:47:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:47:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:47:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:47:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:47:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:47:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:47:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:47:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:47:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:47:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:47:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:47:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:47:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:47:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:47:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:47:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:47:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:47:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:47:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:47:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:47:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:47:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:47:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:47:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:47:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:47:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:47:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=10459], [object_number=74, tag_tid=14, entry_tid=10458], [object_number=73, tag_tid=14, entry_tid=10457], [object_number=72, tag_tid=14, entry_tid=10456]]], state=connected] 2024-04-24T00:48:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=10459], [object_number=74, tag_tid=14, entry_tid=10458], [object_number=73, tag_tid=14, entry_tid=10457], [object_number=72, tag_tid=14, entry_tid=10456]]], state=connected] 2024-04-24T00:48:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=75, tag_tid=14, entry_tid=10459' 2024-04-24T00:48:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=75, tag_tid=14, entry_tid=10459' 2024-04-24T00:48:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=10459' '!=' 'object_number=75, tag_tid=14, entry_tid=10287' 2024-04-24T00:48:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=10459' = 'object_number=75, tag_tid=14, entry_tid=10287' 2024-04-24T00:48:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=75, tag_tid=14, entry_tid=10459' 2024-04-24T00:48:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=10459' 2024-04-24T00:48:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=10459' 2024-04-24T00:48:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10459 2024-04-24T00:48:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 10459 2024-04-24T00:48:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=10653], [object_number=72, tag_tid=14, entry_tid=10652], [object_number=75, tag_tid=14, entry_tid=10651], [object_number=74, tag_tid=14, entry_tid=10650]]], state=connected] 2024-04-24T00:48:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=10653], [object_number=72, tag_tid=14, entry_tid=10652], [object_number=75, tag_tid=14, entry_tid=10651], [object_number=74, tag_tid=14, entry_tid=10650]]], state=connected] 2024-04-24T00:48:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=73, tag_tid=14, entry_tid=10653' 2024-04-24T00:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=14, entry_tid=10653' 2024-04-24T00:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=10653' '!=' 'object_number=75, tag_tid=14, entry_tid=10459' 2024-04-24T00:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=10653' = 'object_number=75, tag_tid=14, entry_tid=10459' 2024-04-24T00:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=73, tag_tid=14, entry_tid=10653' 2024-04-24T00:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=10653' 2024-04-24T00:48:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=10653' 2024-04-24T00:48:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10653 2024-04-24T00:48:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 10653 2024-04-24T00:48:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:01.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:01.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:01.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:01.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:01.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=10831], [object_number=74, tag_tid=14, entry_tid=10830], [object_number=73, tag_tid=14, entry_tid=10829], [object_number=72, tag_tid=14, entry_tid=10828]]], state=connected] 2024-04-24T00:48:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=10831], [object_number=74, tag_tid=14, entry_tid=10830], [object_number=73, tag_tid=14, entry_tid=10829], [object_number=72, tag_tid=14, entry_tid=10828]]], state=connected] 2024-04-24T00:48:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=75, tag_tid=14, entry_tid=10831' 2024-04-24T00:48:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=75, tag_tid=14, entry_tid=10831' 2024-04-24T00:48:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=10831' '!=' 'object_number=73, tag_tid=14, entry_tid=10653' 2024-04-24T00:48:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=10831' = 'object_number=73, tag_tid=14, entry_tid=10653' 2024-04-24T00:48:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=75, tag_tid=14, entry_tid=10831' 2024-04-24T00:48:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=10831' 2024-04-24T00:48:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=10831' 2024-04-24T00:48:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10831 2024-04-24T00:48:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 10831 2024-04-24T00:48:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=10995], [object_number=74, tag_tid=14, entry_tid=10994], [object_number=73, tag_tid=14, entry_tid=10993], [object_number=72, tag_tid=14, entry_tid=10992]]], state=connected] 2024-04-24T00:48:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:02.660 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:48:02.663+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db2616000 2024-04-24T00:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:02.663 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:48:02.663+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc0178680 2024-04-24T00:48:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=75, tag_tid=14, entry_tid=10995], [object_number=74, tag_tid=14, entry_tid=10994], [object_number=73, tag_tid=14, entry_tid=10993], [object_number=72, tag_tid=14, entry_tid=10992]]], state=connected] 2024-04-24T00:48:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=75, tag_tid=14, entry_tid=10995' 2024-04-24T00:48:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=75, tag_tid=14, entry_tid=10995' 2024-04-24T00:48:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=10995' '!=' 'object_number=75, tag_tid=14, entry_tid=10831' 2024-04-24T00:48:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=75, tag_tid=14, entry_tid=10995' = 'object_number=75, tag_tid=14, entry_tid=10831' 2024-04-24T00:48:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=75, tag_tid=14, entry_tid=10995' 2024-04-24T00:48:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=10995' 2024-04-24T00:48:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=75, tag_tid=14, entry_tid=10995' 2024-04-24T00:48:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10995 2024-04-24T00:48:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 10995 2024-04-24T00:48:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=74, tag_tid=14, entry_tid=11182], [object_number=73, tag_tid=14, entry_tid=11181], [object_number=72, tag_tid=14, entry_tid=11180], [object_number=75, tag_tid=14, entry_tid=11179]]], state=connected] 2024-04-24T00:48:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=74, tag_tid=14, entry_tid=11182], [object_number=73, tag_tid=14, entry_tid=11181], [object_number=72, tag_tid=14, entry_tid=11180], [object_number=75, tag_tid=14, entry_tid=11179]]], state=connected] 2024-04-24T00:48:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=74, tag_tid=14, entry_tid=11182' 2024-04-24T00:48:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=74, tag_tid=14, entry_tid=11182' 2024-04-24T00:48:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=74, tag_tid=14, entry_tid=11182' '!=' 'object_number=75, tag_tid=14, entry_tid=10995' 2024-04-24T00:48:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=74, tag_tid=14, entry_tid=11182' = 'object_number=75, tag_tid=14, entry_tid=10995' 2024-04-24T00:48:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=74, tag_tid=14, entry_tid=11182' 2024-04-24T00:48:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=74, tag_tid=14, entry_tid=11182' 2024-04-24T00:48:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=74, tag_tid=14, entry_tid=11182' 2024-04-24T00:48:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11182 2024-04-24T00:48:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 11182 2024-04-24T00:48:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=11353], [object_number=72, tag_tid=14, entry_tid=11352], [object_number=75, tag_tid=14, entry_tid=11351], [object_number=74, tag_tid=14, entry_tid=11350]]], state=connected] 2024-04-24T00:48:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=11353], [object_number=72, tag_tid=14, entry_tid=11352], [object_number=75, tag_tid=14, entry_tid=11351], [object_number=74, tag_tid=14, entry_tid=11350]]], state=connected] 2024-04-24T00:48:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=73, tag_tid=14, entry_tid=11353' 2024-04-24T00:48:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=14, entry_tid=11353' 2024-04-24T00:48:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=11353' '!=' 'object_number=74, tag_tid=14, entry_tid=11182' 2024-04-24T00:48:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=11353' = 'object_number=74, tag_tid=14, entry_tid=11182' 2024-04-24T00:48:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=73, tag_tid=14, entry_tid=11353' 2024-04-24T00:48:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=11353' 2024-04-24T00:48:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=11353' 2024-04-24T00:48:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11353 2024-04-24T00:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 11353 2024-04-24T00:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=11521], [object_number=72, tag_tid=14, entry_tid=11520], [object_number=75, tag_tid=14, entry_tid=11519], [object_number=74, tag_tid=14, entry_tid=11518]]], state=connected] 2024-04-24T00:48:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=11521], [object_number=72, tag_tid=14, entry_tid=11520], [object_number=75, tag_tid=14, entry_tid=11519], [object_number=74, tag_tid=14, entry_tid=11518]]], state=connected] 2024-04-24T00:48:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=73, tag_tid=14, entry_tid=11521' 2024-04-24T00:48:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=14, entry_tid=11521' 2024-04-24T00:48:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=11521' '!=' 'object_number=73, tag_tid=14, entry_tid=11353' 2024-04-24T00:48:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=11521' = 'object_number=73, tag_tid=14, entry_tid=11353' 2024-04-24T00:48:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=73, tag_tid=14, entry_tid=11521' 2024-04-24T00:48:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=11521' 2024-04-24T00:48:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=11521' 2024-04-24T00:48:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11521 2024-04-24T00:48:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 11521 2024-04-24T00:48:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=11692], [object_number=75, tag_tid=14, entry_tid=11691], [object_number=74, tag_tid=14, entry_tid=11690], [object_number=73, tag_tid=14, entry_tid=11689]]], state=connected] 2024-04-24T00:48:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=72, tag_tid=14, entry_tid=11692], [object_number=75, tag_tid=14, entry_tid=11691], [object_number=74, tag_tid=14, entry_tid=11690], [object_number=73, tag_tid=14, entry_tid=11689]]], state=connected] 2024-04-24T00:48:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=72, tag_tid=14, entry_tid=11692' 2024-04-24T00:48:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=72, tag_tid=14, entry_tid=11692' 2024-04-24T00:48:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=11692' '!=' 'object_number=73, tag_tid=14, entry_tid=11521' 2024-04-24T00:48:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=72, tag_tid=14, entry_tid=11692' = 'object_number=73, tag_tid=14, entry_tid=11521' 2024-04-24T00:48:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=72, tag_tid=14, entry_tid=11692' 2024-04-24T00:48:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=11692' 2024-04-24T00:48:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=72, tag_tid=14, entry_tid=11692' 2024-04-24T00:48:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11692 2024-04-24T00:48:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 11692 2024-04-24T00:48:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=11877], [object_number=72, tag_tid=14, entry_tid=11876], [object_number=75, tag_tid=14, entry_tid=11875], [object_number=74, tag_tid=14, entry_tid=11874]]], state=connected] 2024-04-24T00:48:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 18 2024-04-24T00:48:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=73, tag_tid=14, entry_tid=11877], [object_number=72, tag_tid=14, entry_tid=11876], [object_number=75, tag_tid=14, entry_tid=11875], [object_number=74, tag_tid=14, entry_tid=11874]]], state=connected] 2024-04-24T00:48:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=73, tag_tid=14, entry_tid=11877' 2024-04-24T00:48:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=14, entry_tid=11877' 2024-04-24T00:48:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=11877' '!=' 'object_number=72, tag_tid=14, entry_tid=11692' 2024-04-24T00:48:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=73, tag_tid=14, entry_tid=11877' = 'object_number=72, tag_tid=14, entry_tid=11692' 2024-04-24T00:48:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=73, tag_tid=14, entry_tid=11877' 2024-04-24T00:48:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=11877' 2024-04-24T00:48:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=73, tag_tid=14, entry_tid=11877' 2024-04-24T00:48:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11877 2024-04-24T00:48:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 11877 2024-04-24T00:48:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=77, tag_tid=14, entry_tid=12017], [object_number=76, tag_tid=14, entry_tid=12016], [object_number=79, tag_tid=14, entry_tid=12015], [object_number=78, tag_tid=14, entry_tid=12014]]], state=connected] 2024-04-24T00:48:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=77, tag_tid=14, entry_tid=12017], [object_number=76, tag_tid=14, entry_tid=12016], [object_number=79, tag_tid=14, entry_tid=12015], [object_number=78, tag_tid=14, entry_tid=12014]]], state=connected] 2024-04-24T00:48:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=77, tag_tid=14, entry_tid=12017' 2024-04-24T00:48:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=77, tag_tid=14, entry_tid=12017' 2024-04-24T00:48:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=77, tag_tid=14, entry_tid=12017' '!=' 'object_number=73, tag_tid=14, entry_tid=11877' 2024-04-24T00:48:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=77, tag_tid=14, entry_tid=12017' = 'object_number=73, tag_tid=14, entry_tid=11877' 2024-04-24T00:48:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=77, tag_tid=14, entry_tid=12017' 2024-04-24T00:48:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=77, tag_tid=14, entry_tid=12017' 2024-04-24T00:48:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=77, tag_tid=14, entry_tid=12017' 2024-04-24T00:48:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12017 2024-04-24T00:48:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 12017 2024-04-24T00:48:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=77, tag_tid=14, entry_tid=12181], [object_number=76, tag_tid=14, entry_tid=12180], [object_number=79, tag_tid=14, entry_tid=12179], [object_number=78, tag_tid=14, entry_tid=12178]]], state=connected] 2024-04-24T00:48:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=77, tag_tid=14, entry_tid=12181], [object_number=76, tag_tid=14, entry_tid=12180], [object_number=79, tag_tid=14, entry_tid=12179], [object_number=78, tag_tid=14, entry_tid=12178]]], state=connected] 2024-04-24T00:48:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=77, tag_tid=14, entry_tid=12181' 2024-04-24T00:48:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=77, tag_tid=14, entry_tid=12181' 2024-04-24T00:48:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=77, tag_tid=14, entry_tid=12181' '!=' 'object_number=77, tag_tid=14, entry_tid=12017' 2024-04-24T00:48:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=77, tag_tid=14, entry_tid=12181' = 'object_number=77, tag_tid=14, entry_tid=12017' 2024-04-24T00:48:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=77, tag_tid=14, entry_tid=12181' 2024-04-24T00:48:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=77, tag_tid=14, entry_tid=12181' 2024-04-24T00:48:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=77, tag_tid=14, entry_tid=12181' 2024-04-24T00:48:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12181 2024-04-24T00:48:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 12181 2024-04-24T00:48:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=12334], [object_number=77, tag_tid=14, entry_tid=12333], [object_number=76, tag_tid=14, entry_tid=12332], [object_number=79, tag_tid=14, entry_tid=12331]]], state=connected] 2024-04-24T00:48:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=12334], [object_number=77, tag_tid=14, entry_tid=12333], [object_number=76, tag_tid=14, entry_tid=12332], [object_number=79, tag_tid=14, entry_tid=12331]]], state=connected] 2024-04-24T00:48:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=78, tag_tid=14, entry_tid=12334' 2024-04-24T00:48:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=78, tag_tid=14, entry_tid=12334' 2024-04-24T00:48:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=12334' '!=' 'object_number=77, tag_tid=14, entry_tid=12181' 2024-04-24T00:48:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=12334' = 'object_number=77, tag_tid=14, entry_tid=12181' 2024-04-24T00:48:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=78, tag_tid=14, entry_tid=12334' 2024-04-24T00:48:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=12334' 2024-04-24T00:48:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=12334' 2024-04-24T00:48:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12334 2024-04-24T00:48:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 12334 2024-04-24T00:48:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:10.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:10.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:10.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:10.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:10.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=12506], [object_number=77, tag_tid=14, entry_tid=12505], [object_number=76, tag_tid=14, entry_tid=12504], [object_number=79, tag_tid=14, entry_tid=12503]]], state=connected] 2024-04-24T00:48:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=12506], [object_number=77, tag_tid=14, entry_tid=12505], [object_number=76, tag_tid=14, entry_tid=12504], [object_number=79, tag_tid=14, entry_tid=12503]]], state=connected] 2024-04-24T00:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=78, tag_tid=14, entry_tid=12506' 2024-04-24T00:48:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=78, tag_tid=14, entry_tid=12506' 2024-04-24T00:48:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=12506' '!=' 'object_number=78, tag_tid=14, entry_tid=12334' 2024-04-24T00:48:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=12506' = 'object_number=78, tag_tid=14, entry_tid=12334' 2024-04-24T00:48:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=78, tag_tid=14, entry_tid=12506' 2024-04-24T00:48:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=12506' 2024-04-24T00:48:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=12506' 2024-04-24T00:48:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12506 2024-04-24T00:48:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 12506 2024-04-24T00:48:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=12758], [object_number=77, tag_tid=14, entry_tid=12757], [object_number=76, tag_tid=14, entry_tid=12756], [object_number=79, tag_tid=14, entry_tid=12755]]], state=connected] 2024-04-24T00:48:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=12758], [object_number=77, tag_tid=14, entry_tid=12757], [object_number=76, tag_tid=14, entry_tid=12756], [object_number=79, tag_tid=14, entry_tid=12755]]], state=connected] 2024-04-24T00:48:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=78, tag_tid=14, entry_tid=12758' 2024-04-24T00:48:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=78, tag_tid=14, entry_tid=12758' 2024-04-24T00:48:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=12758' '!=' 'object_number=78, tag_tid=14, entry_tid=12506' 2024-04-24T00:48:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=12758' = 'object_number=78, tag_tid=14, entry_tid=12506' 2024-04-24T00:48:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=78, tag_tid=14, entry_tid=12758' 2024-04-24T00:48:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=12758' 2024-04-24T00:48:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=12758' 2024-04-24T00:48:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12758 2024-04-24T00:48:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 12758 2024-04-24T00:48:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=12931], [object_number=78, tag_tid=14, entry_tid=12930], [object_number=77, tag_tid=14, entry_tid=12929], [object_number=76, tag_tid=14, entry_tid=12928]]], state=connected] 2024-04-24T00:48:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=12931], [object_number=78, tag_tid=14, entry_tid=12930], [object_number=77, tag_tid=14, entry_tid=12929], [object_number=76, tag_tid=14, entry_tid=12928]]], state=connected] 2024-04-24T00:48:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=79, tag_tid=14, entry_tid=12931' 2024-04-24T00:48:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=79, tag_tid=14, entry_tid=12931' 2024-04-24T00:48:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=12931' '!=' 'object_number=78, tag_tid=14, entry_tid=12758' 2024-04-24T00:48:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=12931' = 'object_number=78, tag_tid=14, entry_tid=12758' 2024-04-24T00:48:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=79, tag_tid=14, entry_tid=12931' 2024-04-24T00:48:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=12931' 2024-04-24T00:48:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=12931' 2024-04-24T00:48:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12931 2024-04-24T00:48:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 12931 2024-04-24T00:48:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=13108], [object_number=79, tag_tid=14, entry_tid=13107], [object_number=78, tag_tid=14, entry_tid=13106], [object_number=77, tag_tid=14, entry_tid=13105]]], state=connected] 2024-04-24T00:48:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=13108], [object_number=79, tag_tid=14, entry_tid=13107], [object_number=78, tag_tid=14, entry_tid=13106], [object_number=77, tag_tid=14, entry_tid=13105]]], state=connected] 2024-04-24T00:48:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=76, tag_tid=14, entry_tid=13108' 2024-04-24T00:48:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=76, tag_tid=14, entry_tid=13108' 2024-04-24T00:48:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=13108' '!=' 'object_number=79, tag_tid=14, entry_tid=12931' 2024-04-24T00:48:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=13108' = 'object_number=79, tag_tid=14, entry_tid=12931' 2024-04-24T00:48:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=76, tag_tid=14, entry_tid=13108' 2024-04-24T00:48:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=13108' 2024-04-24T00:48:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=13108' 2024-04-24T00:48:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13108 2024-04-24T00:48:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 13108 2024-04-24T00:48:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=13276], [object_number=79, tag_tid=14, entry_tid=13275], [object_number=78, tag_tid=14, entry_tid=13274], [object_number=77, tag_tid=14, entry_tid=13273]]], state=connected] 2024-04-24T00:48:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=13276], [object_number=79, tag_tid=14, entry_tid=13275], [object_number=78, tag_tid=14, entry_tid=13274], [object_number=77, tag_tid=14, entry_tid=13273]]], state=connected] 2024-04-24T00:48:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=76, tag_tid=14, entry_tid=13276' 2024-04-24T00:48:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=76, tag_tid=14, entry_tid=13276' 2024-04-24T00:48:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=13276' '!=' 'object_number=76, tag_tid=14, entry_tid=13108' 2024-04-24T00:48:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=13276' = 'object_number=76, tag_tid=14, entry_tid=13108' 2024-04-24T00:48:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=76, tag_tid=14, entry_tid=13276' 2024-04-24T00:48:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=13276' 2024-04-24T00:48:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=13276' 2024-04-24T00:48:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13276 2024-04-24T00:48:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 13276 2024-04-24T00:48:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:14.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=13431], [object_number=78, tag_tid=14, entry_tid=13430], [object_number=77, tag_tid=14, entry_tid=13429], [object_number=76, tag_tid=14, entry_tid=13428]]], state=connected] 2024-04-24T00:48:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=13431], [object_number=78, tag_tid=14, entry_tid=13430], [object_number=77, tag_tid=14, entry_tid=13429], [object_number=76, tag_tid=14, entry_tid=13428]]], state=connected] 2024-04-24T00:48:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=79, tag_tid=14, entry_tid=13431' 2024-04-24T00:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=79, tag_tid=14, entry_tid=13431' 2024-04-24T00:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=13431' '!=' 'object_number=76, tag_tid=14, entry_tid=13276' 2024-04-24T00:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=13431' = 'object_number=76, tag_tid=14, entry_tid=13276' 2024-04-24T00:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=79, tag_tid=14, entry_tid=13431' 2024-04-24T00:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=13431' 2024-04-24T00:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:15.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=13431' 2024-04-24T00:48:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13431 2024-04-24T00:48:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 13431 2024-04-24T00:48:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=13588], [object_number=79, tag_tid=14, entry_tid=13587], [object_number=78, tag_tid=14, entry_tid=13586], [object_number=77, tag_tid=14, entry_tid=13585]]], state=connected] 2024-04-24T00:48:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=13588], [object_number=79, tag_tid=14, entry_tid=13587], [object_number=78, tag_tid=14, entry_tid=13586], [object_number=77, tag_tid=14, entry_tid=13585]]], state=connected] 2024-04-24T00:48:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=76, tag_tid=14, entry_tid=13588' 2024-04-24T00:48:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=76, tag_tid=14, entry_tid=13588' 2024-04-24T00:48:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=13588' '!=' 'object_number=79, tag_tid=14, entry_tid=13431' 2024-04-24T00:48:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=13588' = 'object_number=79, tag_tid=14, entry_tid=13431' 2024-04-24T00:48:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=76, tag_tid=14, entry_tid=13588' 2024-04-24T00:48:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=13588' 2024-04-24T00:48:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=13588' 2024-04-24T00:48:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13588 2024-04-24T00:48:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 13588 2024-04-24T00:48:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=13746], [object_number=77, tag_tid=14, entry_tid=13745], [object_number=76, tag_tid=14, entry_tid=13744], [object_number=79, tag_tid=14, entry_tid=13743]]], state=connected] 2024-04-24T00:48:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=13746], [object_number=77, tag_tid=14, entry_tid=13745], [object_number=76, tag_tid=14, entry_tid=13744], [object_number=79, tag_tid=14, entry_tid=13743]]], state=connected] 2024-04-24T00:48:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=78, tag_tid=14, entry_tid=13746' 2024-04-24T00:48:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=78, tag_tid=14, entry_tid=13746' 2024-04-24T00:48:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=13746' '!=' 'object_number=76, tag_tid=14, entry_tid=13588' 2024-04-24T00:48:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=13746' = 'object_number=76, tag_tid=14, entry_tid=13588' 2024-04-24T00:48:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=78, tag_tid=14, entry_tid=13746' 2024-04-24T00:48:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=13746' 2024-04-24T00:48:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=13746' 2024-04-24T00:48:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13746 2024-04-24T00:48:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 13746 2024-04-24T00:48:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:17.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=13911], [object_number=78, tag_tid=14, entry_tid=13910], [object_number=77, tag_tid=14, entry_tid=13909], [object_number=76, tag_tid=14, entry_tid=13908]]], state=connected] 2024-04-24T00:48:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=13911], [object_number=78, tag_tid=14, entry_tid=13910], [object_number=77, tag_tid=14, entry_tid=13909], [object_number=76, tag_tid=14, entry_tid=13908]]], state=connected] 2024-04-24T00:48:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=79, tag_tid=14, entry_tid=13911' 2024-04-24T00:48:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=79, tag_tid=14, entry_tid=13911' 2024-04-24T00:48:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=13911' '!=' 'object_number=78, tag_tid=14, entry_tid=13746' 2024-04-24T00:48:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=13911' = 'object_number=78, tag_tid=14, entry_tid=13746' 2024-04-24T00:48:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=79, tag_tid=14, entry_tid=13911' 2024-04-24T00:48:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=13911' 2024-04-24T00:48:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=13911' 2024-04-24T00:48:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13911 2024-04-24T00:48:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 13911 2024-04-24T00:48:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=14082], [object_number=77, tag_tid=14, entry_tid=14081], [object_number=76, tag_tid=14, entry_tid=14080], [object_number=79, tag_tid=14, entry_tid=14079]]], state=connected] 2024-04-24T00:48:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=14082], [object_number=77, tag_tid=14, entry_tid=14081], [object_number=76, tag_tid=14, entry_tid=14080], [object_number=79, tag_tid=14, entry_tid=14079]]], state=connected] 2024-04-24T00:48:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=78, tag_tid=14, entry_tid=14082' 2024-04-24T00:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=78, tag_tid=14, entry_tid=14082' 2024-04-24T00:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=14082' '!=' 'object_number=79, tag_tid=14, entry_tid=13911' 2024-04-24T00:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=14082' = 'object_number=79, tag_tid=14, entry_tid=13911' 2024-04-24T00:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=78, tag_tid=14, entry_tid=14082' 2024-04-24T00:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=14082' 2024-04-24T00:48:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=14082' 2024-04-24T00:48:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14082 2024-04-24T00:48:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 14082 2024-04-24T00:48:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=14251], [object_number=78, tag_tid=14, entry_tid=14250], [object_number=77, tag_tid=14, entry_tid=14249], [object_number=76, tag_tid=14, entry_tid=14248]]], state=connected] 2024-04-24T00:48:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=14251], [object_number=78, tag_tid=14, entry_tid=14250], [object_number=77, tag_tid=14, entry_tid=14249], [object_number=76, tag_tid=14, entry_tid=14248]]], state=connected] 2024-04-24T00:48:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=79, tag_tid=14, entry_tid=14251' 2024-04-24T00:48:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=79, tag_tid=14, entry_tid=14251' 2024-04-24T00:48:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=14251' '!=' 'object_number=78, tag_tid=14, entry_tid=14082' 2024-04-24T00:48:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=14251' = 'object_number=78, tag_tid=14, entry_tid=14082' 2024-04-24T00:48:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=79, tag_tid=14, entry_tid=14251' 2024-04-24T00:48:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=14251' 2024-04-24T00:48:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=14251' 2024-04-24T00:48:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14251 2024-04-24T00:48:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 14251 2024-04-24T00:48:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=14404], [object_number=79, tag_tid=14, entry_tid=14403], [object_number=78, tag_tid=14, entry_tid=14402], [object_number=77, tag_tid=14, entry_tid=14401]]], state=connected] 2024-04-24T00:48:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=14404], [object_number=79, tag_tid=14, entry_tid=14403], [object_number=78, tag_tid=14, entry_tid=14402], [object_number=77, tag_tid=14, entry_tid=14401]]], state=connected] 2024-04-24T00:48:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=76, tag_tid=14, entry_tid=14404' 2024-04-24T00:48:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=76, tag_tid=14, entry_tid=14404' 2024-04-24T00:48:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=14404' '!=' 'object_number=79, tag_tid=14, entry_tid=14251' 2024-04-24T00:48:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=14404' = 'object_number=79, tag_tid=14, entry_tid=14251' 2024-04-24T00:48:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=76, tag_tid=14, entry_tid=14404' 2024-04-24T00:48:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=14404' 2024-04-24T00:48:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=14404' 2024-04-24T00:48:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14404 2024-04-24T00:48:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 14404 2024-04-24T00:48:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=14563], [object_number=78, tag_tid=14, entry_tid=14562], [object_number=77, tag_tid=14, entry_tid=14561], [object_number=76, tag_tid=14, entry_tid=14560]]], state=connected] 2024-04-24T00:48:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=14563], [object_number=78, tag_tid=14, entry_tid=14562], [object_number=77, tag_tid=14, entry_tid=14561], [object_number=76, tag_tid=14, entry_tid=14560]]], state=connected] 2024-04-24T00:48:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=79, tag_tid=14, entry_tid=14563' 2024-04-24T00:48:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=79, tag_tid=14, entry_tid=14563' 2024-04-24T00:48:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=14563' '!=' 'object_number=76, tag_tid=14, entry_tid=14404' 2024-04-24T00:48:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=14563' = 'object_number=76, tag_tid=14, entry_tid=14404' 2024-04-24T00:48:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=79, tag_tid=14, entry_tid=14563' 2024-04-24T00:48:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=14563' 2024-04-24T00:48:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=14563' 2024-04-24T00:48:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14563 2024-04-24T00:48:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 14563 2024-04-24T00:48:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=14736], [object_number=79, tag_tid=14, entry_tid=14735], [object_number=78, tag_tid=14, entry_tid=14734], [object_number=77, tag_tid=14, entry_tid=14733]]], state=connected] 2024-04-24T00:48:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=14736], [object_number=79, tag_tid=14, entry_tid=14735], [object_number=78, tag_tid=14, entry_tid=14734], [object_number=77, tag_tid=14, entry_tid=14733]]], state=connected] 2024-04-24T00:48:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=76, tag_tid=14, entry_tid=14736' 2024-04-24T00:48:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=76, tag_tid=14, entry_tid=14736' 2024-04-24T00:48:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=14736' '!=' 'object_number=79, tag_tid=14, entry_tid=14563' 2024-04-24T00:48:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=14736' = 'object_number=79, tag_tid=14, entry_tid=14563' 2024-04-24T00:48:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=76, tag_tid=14, entry_tid=14736' 2024-04-24T00:48:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=14736' 2024-04-24T00:48:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=14736' 2024-04-24T00:48:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14736 2024-04-24T00:48:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 14736 2024-04-24T00:48:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=14904], [object_number=79, tag_tid=14, entry_tid=14903], [object_number=78, tag_tid=14, entry_tid=14902], [object_number=77, tag_tid=14, entry_tid=14901]]], state=connected] 2024-04-24T00:48:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=76, tag_tid=14, entry_tid=14904], [object_number=79, tag_tid=14, entry_tid=14903], [object_number=78, tag_tid=14, entry_tid=14902], [object_number=77, tag_tid=14, entry_tid=14901]]], state=connected] 2024-04-24T00:48:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=76, tag_tid=14, entry_tid=14904' 2024-04-24T00:48:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=76, tag_tid=14, entry_tid=14904' 2024-04-24T00:48:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=14904' '!=' 'object_number=76, tag_tid=14, entry_tid=14736' 2024-04-24T00:48:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=76, tag_tid=14, entry_tid=14904' = 'object_number=76, tag_tid=14, entry_tid=14736' 2024-04-24T00:48:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=76, tag_tid=14, entry_tid=14904' 2024-04-24T00:48:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=14904' 2024-04-24T00:48:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=76, tag_tid=14, entry_tid=14904' 2024-04-24T00:48:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14904 2024-04-24T00:48:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 14904 2024-04-24T00:48:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=15075], [object_number=78, tag_tid=14, entry_tid=15074], [object_number=77, tag_tid=14, entry_tid=15073], [object_number=76, tag_tid=14, entry_tid=15072]]], state=connected] 2024-04-24T00:48:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=15075], [object_number=78, tag_tid=14, entry_tid=15074], [object_number=77, tag_tid=14, entry_tid=15073], [object_number=76, tag_tid=14, entry_tid=15072]]], state=connected] 2024-04-24T00:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=79, tag_tid=14, entry_tid=15075' 2024-04-24T00:48:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=79, tag_tid=14, entry_tid=15075' 2024-04-24T00:48:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=15075' '!=' 'object_number=76, tag_tid=14, entry_tid=14904' 2024-04-24T00:48:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=15075' = 'object_number=76, tag_tid=14, entry_tid=14904' 2024-04-24T00:48:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=79, tag_tid=14, entry_tid=15075' 2024-04-24T00:48:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=15075' 2024-04-24T00:48:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=15075' 2024-04-24T00:48:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15075 2024-04-24T00:48:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 15075 2024-04-24T00:48:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=15243], [object_number=78, tag_tid=14, entry_tid=15242], [object_number=77, tag_tid=14, entry_tid=15241], [object_number=76, tag_tid=14, entry_tid=15240]]], state=connected] 2024-04-24T00:48:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=15243], [object_number=78, tag_tid=14, entry_tid=15242], [object_number=77, tag_tid=14, entry_tid=15241], [object_number=76, tag_tid=14, entry_tid=15240]]], state=connected] 2024-04-24T00:48:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=79, tag_tid=14, entry_tid=15243' 2024-04-24T00:48:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=79, tag_tid=14, entry_tid=15243' 2024-04-24T00:48:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=15243' '!=' 'object_number=79, tag_tid=14, entry_tid=15075' 2024-04-24T00:48:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=15243' = 'object_number=79, tag_tid=14, entry_tid=15075' 2024-04-24T00:48:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=79, tag_tid=14, entry_tid=15243' 2024-04-24T00:48:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=15243' 2024-04-24T00:48:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=15243' 2024-04-24T00:48:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15243 2024-04-24T00:48:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 15243 2024-04-24T00:48:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=15394], [object_number=77, tag_tid=14, entry_tid=15393], [object_number=76, tag_tid=14, entry_tid=15392], [object_number=79, tag_tid=14, entry_tid=15391]]], state=connected] 2024-04-24T00:48:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=15394], [object_number=77, tag_tid=14, entry_tid=15393], [object_number=76, tag_tid=14, entry_tid=15392], [object_number=79, tag_tid=14, entry_tid=15391]]], state=connected] 2024-04-24T00:48:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=78, tag_tid=14, entry_tid=15394' 2024-04-24T00:48:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=78, tag_tid=14, entry_tid=15394' 2024-04-24T00:48:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=15394' '!=' 'object_number=79, tag_tid=14, entry_tid=15243' 2024-04-24T00:48:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=15394' = 'object_number=79, tag_tid=14, entry_tid=15243' 2024-04-24T00:48:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=78, tag_tid=14, entry_tid=15394' 2024-04-24T00:48:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=15394' 2024-04-24T00:48:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=15394' 2024-04-24T00:48:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15394 2024-04-24T00:48:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 15394 2024-04-24T00:48:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=77, tag_tid=14, entry_tid=15541], [object_number=76, tag_tid=14, entry_tid=15540], [object_number=79, tag_tid=14, entry_tid=15539], [object_number=78, tag_tid=14, entry_tid=15538]]], state=connected] 2024-04-24T00:48:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=77, tag_tid=14, entry_tid=15541], [object_number=76, tag_tid=14, entry_tid=15540], [object_number=79, tag_tid=14, entry_tid=15539], [object_number=78, tag_tid=14, entry_tid=15538]]], state=connected] 2024-04-24T00:48:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=77, tag_tid=14, entry_tid=15541' 2024-04-24T00:48:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=77, tag_tid=14, entry_tid=15541' 2024-04-24T00:48:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=77, tag_tid=14, entry_tid=15541' '!=' 'object_number=78, tag_tid=14, entry_tid=15394' 2024-04-24T00:48:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=77, tag_tid=14, entry_tid=15541' = 'object_number=78, tag_tid=14, entry_tid=15394' 2024-04-24T00:48:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=77, tag_tid=14, entry_tid=15541' 2024-04-24T00:48:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=77, tag_tid=14, entry_tid=15541' 2024-04-24T00:48:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=77, tag_tid=14, entry_tid=15541' 2024-04-24T00:48:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15541 2024-04-24T00:48:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 15541 2024-04-24T00:48:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=15699], [object_number=78, tag_tid=14, entry_tid=15698], [object_number=77, tag_tid=14, entry_tid=15697], [object_number=76, tag_tid=14, entry_tid=15696]]], state=connected] 2024-04-24T00:48:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=79, tag_tid=14, entry_tid=15699], [object_number=78, tag_tid=14, entry_tid=15698], [object_number=77, tag_tid=14, entry_tid=15697], [object_number=76, tag_tid=14, entry_tid=15696]]], state=connected] 2024-04-24T00:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=79, tag_tid=14, entry_tid=15699' 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=79, tag_tid=14, entry_tid=15699' 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=15699' '!=' 'object_number=77, tag_tid=14, entry_tid=15541' 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=79, tag_tid=14, entry_tid=15699' = 'object_number=77, tag_tid=14, entry_tid=15541' 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=79, tag_tid=14, entry_tid=15699' 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=15699' 2024-04-24T00:48:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=79, tag_tid=14, entry_tid=15699' 2024-04-24T00:48:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15699 2024-04-24T00:48:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 15699 2024-04-24T00:48:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=77, tag_tid=14, entry_tid=15857], [object_number=76, tag_tid=14, entry_tid=15856], [object_number=79, tag_tid=14, entry_tid=15855], [object_number=78, tag_tid=14, entry_tid=15854]]], state=connected] 2024-04-24T00:48:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=77, tag_tid=14, entry_tid=15857], [object_number=76, tag_tid=14, entry_tid=15856], [object_number=79, tag_tid=14, entry_tid=15855], [object_number=78, tag_tid=14, entry_tid=15854]]], state=connected] 2024-04-24T00:48:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=77, tag_tid=14, entry_tid=15857' 2024-04-24T00:48:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=77, tag_tid=14, entry_tid=15857' 2024-04-24T00:48:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=77, tag_tid=14, entry_tid=15857' '!=' 'object_number=79, tag_tid=14, entry_tid=15699' 2024-04-24T00:48:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=77, tag_tid=14, entry_tid=15857' = 'object_number=79, tag_tid=14, entry_tid=15699' 2024-04-24T00:48:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=77, tag_tid=14, entry_tid=15857' 2024-04-24T00:48:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=77, tag_tid=14, entry_tid=15857' 2024-04-24T00:48:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=77, tag_tid=14, entry_tid=15857' 2024-04-24T00:48:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15857 2024-04-24T00:48:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 15857 2024-04-24T00:48:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=16018], [object_number=77, tag_tid=14, entry_tid=16017], [object_number=76, tag_tid=14, entry_tid=16016], [object_number=79, tag_tid=14, entry_tid=16015]]], state=connected] 2024-04-24T00:48:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 19 2024-04-24T00:48:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=78, tag_tid=14, entry_tid=16018], [object_number=77, tag_tid=14, entry_tid=16017], [object_number=76, tag_tid=14, entry_tid=16016], [object_number=79, tag_tid=14, entry_tid=16015]]], state=connected] 2024-04-24T00:48:29.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=78, tag_tid=14, entry_tid=16018' 2024-04-24T00:48:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=78, tag_tid=14, entry_tid=16018' 2024-04-24T00:48:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=16018' '!=' 'object_number=77, tag_tid=14, entry_tid=15857' 2024-04-24T00:48:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=78, tag_tid=14, entry_tid=16018' = 'object_number=77, tag_tid=14, entry_tid=15857' 2024-04-24T00:48:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=78, tag_tid=14, entry_tid=16018' 2024-04-24T00:48:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=16018' 2024-04-24T00:48:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=78, tag_tid=14, entry_tid=16018' 2024-04-24T00:48:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16018 2024-04-24T00:48:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 16018 2024-04-24T00:48:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=81, tag_tid=14, entry_tid=16125], [object_number=80, tag_tid=14, entry_tid=16124], [object_number=83, tag_tid=14, entry_tid=16123], [object_number=82, tag_tid=14, entry_tid=16122]]], state=connected] 2024-04-24T00:48:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=81, tag_tid=14, entry_tid=16125], [object_number=80, tag_tid=14, entry_tid=16124], [object_number=83, tag_tid=14, entry_tid=16123], [object_number=82, tag_tid=14, entry_tid=16122]]], state=connected] 2024-04-24T00:48:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=81, tag_tid=14, entry_tid=16125' 2024-04-24T00:48:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=81, tag_tid=14, entry_tid=16125' 2024-04-24T00:48:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=81, tag_tid=14, entry_tid=16125' '!=' 'object_number=78, tag_tid=14, entry_tid=16018' 2024-04-24T00:48:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=81, tag_tid=14, entry_tid=16125' = 'object_number=78, tag_tid=14, entry_tid=16018' 2024-04-24T00:48:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=81, tag_tid=14, entry_tid=16125' 2024-04-24T00:48:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=81, tag_tid=14, entry_tid=16125' 2024-04-24T00:48:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=81, tag_tid=14, entry_tid=16125' 2024-04-24T00:48:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16125 2024-04-24T00:48:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 16125 2024-04-24T00:48:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=16266], [object_number=81, tag_tid=14, entry_tid=16265], [object_number=80, tag_tid=14, entry_tid=16264], [object_number=83, tag_tid=14, entry_tid=16263]]], state=connected] 2024-04-24T00:48:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=16266], [object_number=81, tag_tid=14, entry_tid=16265], [object_number=80, tag_tid=14, entry_tid=16264], [object_number=83, tag_tid=14, entry_tid=16263]]], state=connected] 2024-04-24T00:48:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=82, tag_tid=14, entry_tid=16266' 2024-04-24T00:48:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=82, tag_tid=14, entry_tid=16266' 2024-04-24T00:48:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=16266' '!=' 'object_number=81, tag_tid=14, entry_tid=16125' 2024-04-24T00:48:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=16266' = 'object_number=81, tag_tid=14, entry_tid=16125' 2024-04-24T00:48:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=82, tag_tid=14, entry_tid=16266' 2024-04-24T00:48:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=16266' 2024-04-24T00:48:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=16266' 2024-04-24T00:48:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16266 2024-04-24T00:48:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 16266 2024-04-24T00:48:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:31.794 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T00:48:31.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T00:48:31.795 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T00:48:31.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T00:48:31.794+0000 7fcc61697640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 62966) UID: 1000 2024-04-24T00:48:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=16467], [object_number=82, tag_tid=14, entry_tid=16466], [object_number=81, tag_tid=14, entry_tid=16465], [object_number=80, tag_tid=14, entry_tid=16464]]], state=connected] 2024-04-24T00:48:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=16467], [object_number=82, tag_tid=14, entry_tid=16466], [object_number=81, tag_tid=14, entry_tid=16465], [object_number=80, tag_tid=14, entry_tid=16464]]], state=connected] 2024-04-24T00:48:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=16467' 2024-04-24T00:48:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=16467' 2024-04-24T00:48:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=16467' '!=' 'object_number=82, tag_tid=14, entry_tid=16266' 2024-04-24T00:48:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=16467' = 'object_number=82, tag_tid=14, entry_tid=16266' 2024-04-24T00:48:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=16467' 2024-04-24T00:48:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=16467' 2024-04-24T00:48:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=16467' 2024-04-24T00:48:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16467 2024-04-24T00:48:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 16467 2024-04-24T00:48:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=16616], [object_number=83, tag_tid=14, entry_tid=16615], [object_number=82, tag_tid=14, entry_tid=16614], [object_number=81, tag_tid=14, entry_tid=16613]]], state=connected] 2024-04-24T00:48:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=16616], [object_number=83, tag_tid=14, entry_tid=16615], [object_number=82, tag_tid=14, entry_tid=16614], [object_number=81, tag_tid=14, entry_tid=16613]]], state=connected] 2024-04-24T00:48:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=16616' 2024-04-24T00:48:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=16616' 2024-04-24T00:48:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=16616' '!=' 'object_number=83, tag_tid=14, entry_tid=16467' 2024-04-24T00:48:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=16616' = 'object_number=83, tag_tid=14, entry_tid=16467' 2024-04-24T00:48:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=16616' 2024-04-24T00:48:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=16616' 2024-04-24T00:48:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=16616' 2024-04-24T00:48:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16616 2024-04-24T00:48:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 16616 2024-04-24T00:48:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:48:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=16764], [object_number=83, tag_tid=14, entry_tid=16763], [object_number=82, tag_tid=14, entry_tid=16762], [object_number=81, tag_tid=14, entry_tid=16761]]], state=connected] 2024-04-24T00:48:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=16764], [object_number=83, tag_tid=14, entry_tid=16763], [object_number=82, tag_tid=14, entry_tid=16762], [object_number=81, tag_tid=14, entry_tid=16761]]], state=connected] 2024-04-24T00:48:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=16764' 2024-04-24T00:48:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=16764' 2024-04-24T00:48:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=16764' '!=' 'object_number=80, tag_tid=14, entry_tid=16616' 2024-04-24T00:48:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=16764' = 'object_number=80, tag_tid=14, entry_tid=16616' 2024-04-24T00:48:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=16764' 2024-04-24T00:48:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=16764' 2024-04-24T00:48:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=16764' 2024-04-24T00:48:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16764 2024-04-24T00:48:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 16764 2024-04-24T00:48:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=16915], [object_number=82, tag_tid=14, entry_tid=16914], [object_number=81, tag_tid=14, entry_tid=16913], [object_number=80, tag_tid=14, entry_tid=16912]]], state=connected] 2024-04-24T00:48:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=16915], [object_number=82, tag_tid=14, entry_tid=16914], [object_number=81, tag_tid=14, entry_tid=16913], [object_number=80, tag_tid=14, entry_tid=16912]]], state=connected] 2024-04-24T00:48:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=16915' 2024-04-24T00:48:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=16915' 2024-04-24T00:48:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=16915' '!=' 'object_number=80, tag_tid=14, entry_tid=16764' 2024-04-24T00:48:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=16915' = 'object_number=80, tag_tid=14, entry_tid=16764' 2024-04-24T00:48:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=16915' 2024-04-24T00:48:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=16915' 2024-04-24T00:48:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=16915' 2024-04-24T00:48:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16915 2024-04-24T00:48:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 16915 2024-04-24T00:48:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:48:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=81, tag_tid=14, entry_tid=17061], [object_number=80, tag_tid=14, entry_tid=17060], [object_number=83, tag_tid=14, entry_tid=17059], [object_number=82, tag_tid=14, entry_tid=17058]]], state=connected] 2024-04-24T00:48:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=81, tag_tid=14, entry_tid=17061], [object_number=80, tag_tid=14, entry_tid=17060], [object_number=83, tag_tid=14, entry_tid=17059], [object_number=82, tag_tid=14, entry_tid=17058]]], state=connected] 2024-04-24T00:48:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=81, tag_tid=14, entry_tid=17061' 2024-04-24T00:48:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=81, tag_tid=14, entry_tid=17061' 2024-04-24T00:48:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=81, tag_tid=14, entry_tid=17061' '!=' 'object_number=83, tag_tid=14, entry_tid=16915' 2024-04-24T00:48:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=81, tag_tid=14, entry_tid=17061' = 'object_number=83, tag_tid=14, entry_tid=16915' 2024-04-24T00:48:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=81, tag_tid=14, entry_tid=17061' 2024-04-24T00:48:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=81, tag_tid=14, entry_tid=17061' 2024-04-24T00:48:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=81, tag_tid=14, entry_tid=17061' 2024-04-24T00:48:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17061 2024-04-24T00:48:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 17061 2024-04-24T00:48:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:48:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=17200], [object_number=83, tag_tid=14, entry_tid=17199], [object_number=82, tag_tid=14, entry_tid=17198], [object_number=81, tag_tid=14, entry_tid=17197]]], state=connected] 2024-04-24T00:48:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=17200], [object_number=83, tag_tid=14, entry_tid=17199], [object_number=82, tag_tid=14, entry_tid=17198], [object_number=81, tag_tid=14, entry_tid=17197]]], state=connected] 2024-04-24T00:48:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=17200' 2024-04-24T00:48:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=17200' 2024-04-24T00:48:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=17200' '!=' 'object_number=81, tag_tid=14, entry_tid=17061' 2024-04-24T00:48:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=17200' = 'object_number=81, tag_tid=14, entry_tid=17061' 2024-04-24T00:48:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=17200' 2024-04-24T00:48:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=17200' 2024-04-24T00:48:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=17200' 2024-04-24T00:48:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17200 2024-04-24T00:48:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 17200 2024-04-24T00:48:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:48:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=17367], [object_number=82, tag_tid=14, entry_tid=17366], [object_number=81, tag_tid=14, entry_tid=17365], [object_number=80, tag_tid=14, entry_tid=17364]]], state=connected] 2024-04-24T00:48:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=17367], [object_number=82, tag_tid=14, entry_tid=17366], [object_number=81, tag_tid=14, entry_tid=17365], [object_number=80, tag_tid=14, entry_tid=17364]]], state=connected] 2024-04-24T00:48:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=17367' 2024-04-24T00:48:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=17367' 2024-04-24T00:48:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=17367' '!=' 'object_number=80, tag_tid=14, entry_tid=17200' 2024-04-24T00:48:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=17367' = 'object_number=80, tag_tid=14, entry_tid=17200' 2024-04-24T00:48:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=17367' 2024-04-24T00:48:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=17367' 2024-04-24T00:48:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=17367' 2024-04-24T00:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17367 2024-04-24T00:48:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 17367 2024-04-24T00:48:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:48:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=17516], [object_number=83, tag_tid=14, entry_tid=17515], [object_number=82, tag_tid=14, entry_tid=17514], [object_number=81, tag_tid=14, entry_tid=17513]]], state=connected] 2024-04-24T00:48:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=17516], [object_number=83, tag_tid=14, entry_tid=17515], [object_number=82, tag_tid=14, entry_tid=17514], [object_number=81, tag_tid=14, entry_tid=17513]]], state=connected] 2024-04-24T00:48:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=17516' 2024-04-24T00:48:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=17516' 2024-04-24T00:48:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=17516' '!=' 'object_number=83, tag_tid=14, entry_tid=17367' 2024-04-24T00:48:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=17516' = 'object_number=83, tag_tid=14, entry_tid=17367' 2024-04-24T00:48:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=17516' 2024-04-24T00:48:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=17516' 2024-04-24T00:48:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=17516' 2024-04-24T00:48:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17516 2024-04-24T00:48:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 17516 2024-04-24T00:48:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:48:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=17656], [object_number=83, tag_tid=14, entry_tid=17655], [object_number=82, tag_tid=14, entry_tid=17654], [object_number=81, tag_tid=14, entry_tid=17653]]], state=connected] 2024-04-24T00:48:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=17656], [object_number=83, tag_tid=14, entry_tid=17655], [object_number=82, tag_tid=14, entry_tid=17654], [object_number=81, tag_tid=14, entry_tid=17653]]], state=connected] 2024-04-24T00:48:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=17656' 2024-04-24T00:48:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=17656' 2024-04-24T00:48:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=17656' '!=' 'object_number=80, tag_tid=14, entry_tid=17516' 2024-04-24T00:48:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=17656' = 'object_number=80, tag_tid=14, entry_tid=17516' 2024-04-24T00:48:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=17656' 2024-04-24T00:48:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=17656' 2024-04-24T00:48:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=17656' 2024-04-24T00:48:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17656 2024-04-24T00:48:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 17656 2024-04-24T00:48:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:48:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:39.795 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T00:48:39.795 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:48:39.795 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T00:48:39.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T00:48:39.796 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T00:48:39.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T00:48:39.798 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-04-24T00:48:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=17810], [object_number=81, tag_tid=14, entry_tid=17809], [object_number=80, tag_tid=14, entry_tid=17808], [object_number=83, tag_tid=14, entry_tid=17807]]], state=connected] 2024-04-24T00:48:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=17810], [object_number=81, tag_tid=14, entry_tid=17809], [object_number=80, tag_tid=14, entry_tid=17808], [object_number=83, tag_tid=14, entry_tid=17807]]], state=connected] 2024-04-24T00:48:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=82, tag_tid=14, entry_tid=17810' 2024-04-24T00:48:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=82, tag_tid=14, entry_tid=17810' 2024-04-24T00:48:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=17810' '!=' 'object_number=80, tag_tid=14, entry_tid=17656' 2024-04-24T00:48:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=17810' = 'object_number=80, tag_tid=14, entry_tid=17656' 2024-04-24T00:48:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=82, tag_tid=14, entry_tid=17810' 2024-04-24T00:48:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=17810' 2024-04-24T00:48:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=17810' 2024-04-24T00:48:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17810 2024-04-24T00:48:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 17810 2024-04-24T00:48:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=17980], [object_number=83, tag_tid=14, entry_tid=17979], [object_number=82, tag_tid=14, entry_tid=17978], [object_number=81, tag_tid=14, entry_tid=17977]]], state=connected] 2024-04-24T00:48:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=17980], [object_number=83, tag_tid=14, entry_tid=17979], [object_number=82, tag_tid=14, entry_tid=17978], [object_number=81, tag_tid=14, entry_tid=17977]]], state=connected] 2024-04-24T00:48:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=17980' 2024-04-24T00:48:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=17980' 2024-04-24T00:48:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=17980' '!=' 'object_number=82, tag_tid=14, entry_tid=17810' 2024-04-24T00:48:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=17980' = 'object_number=82, tag_tid=14, entry_tid=17810' 2024-04-24T00:48:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=17980' 2024-04-24T00:48:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=17980' 2024-04-24T00:48:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=17980' 2024-04-24T00:48:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=17980 2024-04-24T00:48:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 17980 2024-04-24T00:48:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=18144], [object_number=83, tag_tid=14, entry_tid=18143], [object_number=82, tag_tid=14, entry_tid=18142], [object_number=81, tag_tid=14, entry_tid=18141]]], state=connected] 2024-04-24T00:48:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=18144], [object_number=83, tag_tid=14, entry_tid=18143], [object_number=82, tag_tid=14, entry_tid=18142], [object_number=81, tag_tid=14, entry_tid=18141]]], state=connected] 2024-04-24T00:48:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=18144' 2024-04-24T00:48:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=18144' 2024-04-24T00:48:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=18144' '!=' 'object_number=80, tag_tid=14, entry_tid=17980' 2024-04-24T00:48:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=18144' = 'object_number=80, tag_tid=14, entry_tid=17980' 2024-04-24T00:48:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=18144' 2024-04-24T00:48:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=18144' 2024-04-24T00:48:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=18144' 2024-04-24T00:48:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18144 2024-04-24T00:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 18144 2024-04-24T00:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=18263], [object_number=82, tag_tid=14, entry_tid=18262], [object_number=81, tag_tid=14, entry_tid=18261], [object_number=80, tag_tid=14, entry_tid=18260]]], state=connected] 2024-04-24T00:48:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=18263], [object_number=82, tag_tid=14, entry_tid=18262], [object_number=81, tag_tid=14, entry_tid=18261], [object_number=80, tag_tid=14, entry_tid=18260]]], state=connected] 2024-04-24T00:48:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=18263' 2024-04-24T00:48:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=18263' 2024-04-24T00:48:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=18263' '!=' 'object_number=80, tag_tid=14, entry_tid=18144' 2024-04-24T00:48:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=18263' = 'object_number=80, tag_tid=14, entry_tid=18144' 2024-04-24T00:48:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=18263' 2024-04-24T00:48:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=18263' 2024-04-24T00:48:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=18263' 2024-04-24T00:48:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18263 2024-04-24T00:48:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 18263 2024-04-24T00:48:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=18410], [object_number=81, tag_tid=14, entry_tid=18409], [object_number=80, tag_tid=14, entry_tid=18408], [object_number=83, tag_tid=14, entry_tid=18407]]], state=connected] 2024-04-24T00:48:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=18410], [object_number=81, tag_tid=14, entry_tid=18409], [object_number=80, tag_tid=14, entry_tid=18408], [object_number=83, tag_tid=14, entry_tid=18407]]], state=connected] 2024-04-24T00:48:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=82, tag_tid=14, entry_tid=18410' 2024-04-24T00:48:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=82, tag_tid=14, entry_tid=18410' 2024-04-24T00:48:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=18410' '!=' 'object_number=83, tag_tid=14, entry_tid=18263' 2024-04-24T00:48:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=18410' = 'object_number=83, tag_tid=14, entry_tid=18263' 2024-04-24T00:48:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=82, tag_tid=14, entry_tid=18410' 2024-04-24T00:48:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=18410' 2024-04-24T00:48:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=18410' 2024-04-24T00:48:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18410 2024-04-24T00:48:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 18410 2024-04-24T00:48:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=18563], [object_number=82, tag_tid=14, entry_tid=18562], [object_number=81, tag_tid=14, entry_tid=18561], [object_number=80, tag_tid=14, entry_tid=18560]]], state=connected] 2024-04-24T00:48:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=18563], [object_number=82, tag_tid=14, entry_tid=18562], [object_number=81, tag_tid=14, entry_tid=18561], [object_number=80, tag_tid=14, entry_tid=18560]]], state=connected] 2024-04-24T00:48:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=18563' 2024-04-24T00:48:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=18563' 2024-04-24T00:48:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=18563' '!=' 'object_number=82, tag_tid=14, entry_tid=18410' 2024-04-24T00:48:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=18563' = 'object_number=82, tag_tid=14, entry_tid=18410' 2024-04-24T00:48:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=18563' 2024-04-24T00:48:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=18563' 2024-04-24T00:48:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=18563' 2024-04-24T00:48:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18563 2024-04-24T00:48:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 18563 2024-04-24T00:48:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=18727], [object_number=82, tag_tid=14, entry_tid=18726], [object_number=81, tag_tid=14, entry_tid=18725], [object_number=80, tag_tid=14, entry_tid=18724]]], state=connected] 2024-04-24T00:48:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=18727], [object_number=82, tag_tid=14, entry_tid=18726], [object_number=81, tag_tid=14, entry_tid=18725], [object_number=80, tag_tid=14, entry_tid=18724]]], state=connected] 2024-04-24T00:48:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=18727' 2024-04-24T00:48:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=18727' 2024-04-24T00:48:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=18727' '!=' 'object_number=83, tag_tid=14, entry_tid=18563' 2024-04-24T00:48:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=18727' = 'object_number=83, tag_tid=14, entry_tid=18563' 2024-04-24T00:48:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=18727' 2024-04-24T00:48:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=18727' 2024-04-24T00:48:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=18727' 2024-04-24T00:48:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18727 2024-04-24T00:48:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 18727 2024-04-24T00:48:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=18878], [object_number=81, tag_tid=14, entry_tid=18877], [object_number=80, tag_tid=14, entry_tid=18876], [object_number=83, tag_tid=14, entry_tid=18875]]], state=connected] 2024-04-24T00:48:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=18878], [object_number=81, tag_tid=14, entry_tid=18877], [object_number=80, tag_tid=14, entry_tid=18876], [object_number=83, tag_tid=14, entry_tid=18875]]], state=connected] 2024-04-24T00:48:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=82, tag_tid=14, entry_tid=18878' 2024-04-24T00:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=82, tag_tid=14, entry_tid=18878' 2024-04-24T00:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=18878' '!=' 'object_number=83, tag_tid=14, entry_tid=18727' 2024-04-24T00:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=18878' = 'object_number=83, tag_tid=14, entry_tid=18727' 2024-04-24T00:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=82, tag_tid=14, entry_tid=18878' 2024-04-24T00:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=18878' 2024-04-24T00:48:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=18878' 2024-04-24T00:48:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=18878 2024-04-24T00:48:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 18878 2024-04-24T00:48:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=19042], [object_number=81, tag_tid=14, entry_tid=19041], [object_number=80, tag_tid=14, entry_tid=19040], [object_number=83, tag_tid=14, entry_tid=19039]]], state=connected] 2024-04-24T00:48:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=19042], [object_number=81, tag_tid=14, entry_tid=19041], [object_number=80, tag_tid=14, entry_tid=19040], [object_number=83, tag_tid=14, entry_tid=19039]]], state=connected] 2024-04-24T00:48:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=82, tag_tid=14, entry_tid=19042' 2024-04-24T00:48:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=82, tag_tid=14, entry_tid=19042' 2024-04-24T00:48:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=19042' '!=' 'object_number=82, tag_tid=14, entry_tid=18878' 2024-04-24T00:48:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=19042' = 'object_number=82, tag_tid=14, entry_tid=18878' 2024-04-24T00:48:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=82, tag_tid=14, entry_tid=19042' 2024-04-24T00:48:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=19042' 2024-04-24T00:48:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=19042' 2024-04-24T00:48:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19042 2024-04-24T00:48:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 19042 2024-04-24T00:48:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=19215], [object_number=82, tag_tid=14, entry_tid=19214], [object_number=81, tag_tid=14, entry_tid=19213], [object_number=80, tag_tid=14, entry_tid=19212]]], state=connected] 2024-04-24T00:48:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=19215], [object_number=82, tag_tid=14, entry_tid=19214], [object_number=81, tag_tid=14, entry_tid=19213], [object_number=80, tag_tid=14, entry_tid=19212]]], state=connected] 2024-04-24T00:48:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=19215' 2024-04-24T00:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=19215' 2024-04-24T00:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=19215' '!=' 'object_number=82, tag_tid=14, entry_tid=19042' 2024-04-24T00:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=19215' = 'object_number=82, tag_tid=14, entry_tid=19042' 2024-04-24T00:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=19215' 2024-04-24T00:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=19215' 2024-04-24T00:48:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=19215' 2024-04-24T00:48:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19215 2024-04-24T00:48:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 19215 2024-04-24T00:48:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=19375], [object_number=82, tag_tid=14, entry_tid=19374], [object_number=81, tag_tid=14, entry_tid=19373], [object_number=80, tag_tid=14, entry_tid=19372]]], state=connected] 2024-04-24T00:48:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=19375], [object_number=82, tag_tid=14, entry_tid=19374], [object_number=81, tag_tid=14, entry_tid=19373], [object_number=80, tag_tid=14, entry_tid=19372]]], state=connected] 2024-04-24T00:48:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=19375' 2024-04-24T00:48:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=19375' 2024-04-24T00:48:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=19375' '!=' 'object_number=83, tag_tid=14, entry_tid=19215' 2024-04-24T00:48:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=19375' = 'object_number=83, tag_tid=14, entry_tid=19215' 2024-04-24T00:48:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=19375' 2024-04-24T00:48:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=19375' 2024-04-24T00:48:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=19375' 2024-04-24T00:48:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19375 2024-04-24T00:48:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 19375 2024-04-24T00:48:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=19535], [object_number=82, tag_tid=14, entry_tid=19534], [object_number=81, tag_tid=14, entry_tid=19533], [object_number=80, tag_tid=14, entry_tid=19532]]], state=connected] 2024-04-24T00:48:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=19535], [object_number=82, tag_tid=14, entry_tid=19534], [object_number=81, tag_tid=14, entry_tid=19533], [object_number=80, tag_tid=14, entry_tid=19532]]], state=connected] 2024-04-24T00:48:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=19535' 2024-04-24T00:48:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=19535' 2024-04-24T00:48:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=19535' '!=' 'object_number=83, tag_tid=14, entry_tid=19375' 2024-04-24T00:48:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=19535' = 'object_number=83, tag_tid=14, entry_tid=19375' 2024-04-24T00:48:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=19535' 2024-04-24T00:48:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=19535' 2024-04-24T00:48:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=19535' 2024-04-24T00:48:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19535 2024-04-24T00:48:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 19535 2024-04-24T00:48:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=19699], [object_number=82, tag_tid=14, entry_tid=19698], [object_number=81, tag_tid=14, entry_tid=19697], [object_number=80, tag_tid=14, entry_tid=19696]]], state=connected] 2024-04-24T00:48:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=19699], [object_number=82, tag_tid=14, entry_tid=19698], [object_number=81, tag_tid=14, entry_tid=19697], [object_number=80, tag_tid=14, entry_tid=19696]]], state=connected] 2024-04-24T00:48:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=19699' 2024-04-24T00:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=19699' 2024-04-24T00:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=19699' '!=' 'object_number=83, tag_tid=14, entry_tid=19535' 2024-04-24T00:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=19699' = 'object_number=83, tag_tid=14, entry_tid=19535' 2024-04-24T00:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=19699' 2024-04-24T00:48:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=19699' 2024-04-24T00:48:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=19699' 2024-04-24T00:48:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19699 2024-04-24T00:48:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 19699 2024-04-24T00:48:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=19864], [object_number=83, tag_tid=14, entry_tid=19863], [object_number=82, tag_tid=14, entry_tid=19862], [object_number=81, tag_tid=14, entry_tid=19861]]], state=connected] 2024-04-24T00:48:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=19864], [object_number=83, tag_tid=14, entry_tid=19863], [object_number=82, tag_tid=14, entry_tid=19862], [object_number=81, tag_tid=14, entry_tid=19861]]], state=connected] 2024-04-24T00:48:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=19864' 2024-04-24T00:48:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=19864' 2024-04-24T00:48:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=19864' '!=' 'object_number=83, tag_tid=14, entry_tid=19699' 2024-04-24T00:48:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=19864' = 'object_number=83, tag_tid=14, entry_tid=19699' 2024-04-24T00:48:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=19864' 2024-04-24T00:48:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=19864' 2024-04-24T00:48:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=19864' 2024-04-24T00:48:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=19864 2024-04-24T00:48:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 19864 2024-04-24T00:48:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=20030], [object_number=81, tag_tid=14, entry_tid=20029], [object_number=80, tag_tid=14, entry_tid=20028], [object_number=83, tag_tid=14, entry_tid=20027]]], state=connected] 2024-04-24T00:48:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=20030], [object_number=81, tag_tid=14, entry_tid=20029], [object_number=80, tag_tid=14, entry_tid=20028], [object_number=83, tag_tid=14, entry_tid=20027]]], state=connected] 2024-04-24T00:48:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=82, tag_tid=14, entry_tid=20030' 2024-04-24T00:48:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=82, tag_tid=14, entry_tid=20030' 2024-04-24T00:48:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=20030' '!=' 'object_number=80, tag_tid=14, entry_tid=19864' 2024-04-24T00:48:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=20030' = 'object_number=80, tag_tid=14, entry_tid=19864' 2024-04-24T00:48:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=82, tag_tid=14, entry_tid=20030' 2024-04-24T00:48:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=20030' 2024-04-24T00:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=20030' 2024-04-24T00:48:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20030 2024-04-24T00:48:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 20030 2024-04-24T00:48:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=20195], [object_number=82, tag_tid=14, entry_tid=20194], [object_number=81, tag_tid=14, entry_tid=20193], [object_number=80, tag_tid=14, entry_tid=20192]]], state=connected] 2024-04-24T00:48:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 20 2024-04-24T00:48:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=20195], [object_number=82, tag_tid=14, entry_tid=20194], [object_number=81, tag_tid=14, entry_tid=20193], [object_number=80, tag_tid=14, entry_tid=20192]]], state=connected] 2024-04-24T00:48:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=20195' 2024-04-24T00:48:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=20195' 2024-04-24T00:48:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=20195' '!=' 'object_number=82, tag_tid=14, entry_tid=20030' 2024-04-24T00:48:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=20195' = 'object_number=82, tag_tid=14, entry_tid=20030' 2024-04-24T00:48:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=20195' 2024-04-24T00:48:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=20195' 2024-04-24T00:48:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=20195' 2024-04-24T00:48:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20195 2024-04-24T00:48:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 20195 2024-04-24T00:48:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:53.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:53.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:53.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:53.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:53.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=20335], [object_number=86, tag_tid=14, entry_tid=20334], [object_number=85, tag_tid=14, entry_tid=20333], [object_number=84, tag_tid=14, entry_tid=20332]]], state=connected] 2024-04-24T00:48:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=20335], [object_number=86, tag_tid=14, entry_tid=20334], [object_number=85, tag_tid=14, entry_tid=20333], [object_number=84, tag_tid=14, entry_tid=20332]]], state=connected] 2024-04-24T00:48:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=20335' 2024-04-24T00:48:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=20335' 2024-04-24T00:48:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=20335' '!=' 'object_number=83, tag_tid=14, entry_tid=20195' 2024-04-24T00:48:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=20335' = 'object_number=83, tag_tid=14, entry_tid=20195' 2024-04-24T00:48:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=20335' 2024-04-24T00:48:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=20335' 2024-04-24T00:48:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=20335' 2024-04-24T00:48:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20335 2024-04-24T00:48:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 20335 2024-04-24T00:48:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=20502], [object_number=85, tag_tid=14, entry_tid=20501], [object_number=84, tag_tid=14, entry_tid=20500], [object_number=87, tag_tid=14, entry_tid=20499]]], state=connected] 2024-04-24T00:48:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=20502], [object_number=85, tag_tid=14, entry_tid=20501], [object_number=84, tag_tid=14, entry_tid=20500], [object_number=87, tag_tid=14, entry_tid=20499]]], state=connected] 2024-04-24T00:48:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=20502' 2024-04-24T00:48:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=20502' 2024-04-24T00:48:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=20502' '!=' 'object_number=87, tag_tid=14, entry_tid=20335' 2024-04-24T00:48:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=20502' = 'object_number=87, tag_tid=14, entry_tid=20335' 2024-04-24T00:48:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=20502' 2024-04-24T00:48:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=20502' 2024-04-24T00:48:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=20502' 2024-04-24T00:48:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20502 2024-04-24T00:48:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 20502 2024-04-24T00:48:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=20675], [object_number=86, tag_tid=14, entry_tid=20674], [object_number=85, tag_tid=14, entry_tid=20673], [object_number=84, tag_tid=14, entry_tid=20672]]], state=connected] 2024-04-24T00:48:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=20675], [object_number=86, tag_tid=14, entry_tid=20674], [object_number=85, tag_tid=14, entry_tid=20673], [object_number=84, tag_tid=14, entry_tid=20672]]], state=connected] 2024-04-24T00:48:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=20675' 2024-04-24T00:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=20675' 2024-04-24T00:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=20675' '!=' 'object_number=86, tag_tid=14, entry_tid=20502' 2024-04-24T00:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=20675' = 'object_number=86, tag_tid=14, entry_tid=20502' 2024-04-24T00:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=20675' 2024-04-24T00:48:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=20675' 2024-04-24T00:48:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=20675' 2024-04-24T00:48:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20675 2024-04-24T00:48:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 20675 2024-04-24T00:48:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=20844], [object_number=87, tag_tid=14, entry_tid=20843], [object_number=86, tag_tid=14, entry_tid=20842], [object_number=85, tag_tid=14, entry_tid=20841]]], state=connected] 2024-04-24T00:48:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=20844], [object_number=87, tag_tid=14, entry_tid=20843], [object_number=86, tag_tid=14, entry_tid=20842], [object_number=85, tag_tid=14, entry_tid=20841]]], state=connected] 2024-04-24T00:48:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=20844' 2024-04-24T00:48:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=20844' 2024-04-24T00:48:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=20844' '!=' 'object_number=87, tag_tid=14, entry_tid=20675' 2024-04-24T00:48:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=20844' = 'object_number=87, tag_tid=14, entry_tid=20675' 2024-04-24T00:48:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=20844' 2024-04-24T00:48:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=20844' 2024-04-24T00:48:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=20844' 2024-04-24T00:48:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20844 2024-04-24T00:48:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 20844 2024-04-24T00:48:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=20992], [object_number=87, tag_tid=14, entry_tid=20991], [object_number=86, tag_tid=14, entry_tid=20990], [object_number=85, tag_tid=14, entry_tid=20989]]], state=connected] 2024-04-24T00:48:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=20992], [object_number=87, tag_tid=14, entry_tid=20991], [object_number=86, tag_tid=14, entry_tid=20990], [object_number=85, tag_tid=14, entry_tid=20989]]], state=connected] 2024-04-24T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=20992' 2024-04-24T00:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=20992' 2024-04-24T00:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=20992' '!=' 'object_number=84, tag_tid=14, entry_tid=20844' 2024-04-24T00:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=20992' = 'object_number=84, tag_tid=14, entry_tid=20844' 2024-04-24T00:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=20992' 2024-04-24T00:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=20992' 2024-04-24T00:48:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=20992' 2024-04-24T00:48:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=20992 2024-04-24T00:48:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 20992 2024-04-24T00:48:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=21144], [object_number=87, tag_tid=14, entry_tid=21143], [object_number=86, tag_tid=14, entry_tid=21142], [object_number=85, tag_tid=14, entry_tid=21141]]], state=connected] 2024-04-24T00:48:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=21144], [object_number=87, tag_tid=14, entry_tid=21143], [object_number=86, tag_tid=14, entry_tid=21142], [object_number=85, tag_tid=14, entry_tid=21141]]], state=connected] 2024-04-24T00:48:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=21144' 2024-04-24T00:48:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=21144' 2024-04-24T00:48:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=21144' '!=' 'object_number=84, tag_tid=14, entry_tid=20992' 2024-04-24T00:48:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=21144' = 'object_number=84, tag_tid=14, entry_tid=20992' 2024-04-24T00:48:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=21144' 2024-04-24T00:48:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=21144' 2024-04-24T00:48:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=21144' 2024-04-24T00:48:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21144 2024-04-24T00:48:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 21144 2024-04-24T00:48:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:58.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:48:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:48:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:48:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:48:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:48:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:48:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:48:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=21298], [object_number=85, tag_tid=14, entry_tid=21297], [object_number=84, tag_tid=14, entry_tid=21296], [object_number=87, tag_tid=14, entry_tid=21295]]], state=connected] 2024-04-24T00:48:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:48:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:48:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:48:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:48:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=21298], [object_number=85, tag_tid=14, entry_tid=21297], [object_number=84, tag_tid=14, entry_tid=21296], [object_number=87, tag_tid=14, entry_tid=21295]]], state=connected] 2024-04-24T00:48:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:48:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=21298' 2024-04-24T00:48:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=21298' 2024-04-24T00:48:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=21298' '!=' 'object_number=84, tag_tid=14, entry_tid=21144' 2024-04-24T00:48:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:48:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=21298' = 'object_number=84, tag_tid=14, entry_tid=21144' 2024-04-24T00:48:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=21298' 2024-04-24T00:48:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:48:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=21298' 2024-04-24T00:48:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:48:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:48:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:48:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:48:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=21298' 2024-04-24T00:48:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:48:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:48:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21298 2024-04-24T00:48:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 21298 2024-04-24T00:48:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:48:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:48:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:48:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:48:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:48:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:48:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:48:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:48:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:48:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:48:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:48:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:48:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:48:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:48:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:48:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:48:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:48:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:48:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:48:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:48:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:48:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:48:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:48:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:48:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:48:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:48:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:48:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:48:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:48:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:48:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:48:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:48:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:48:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:48:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:48:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:48:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=21441], [object_number=84, tag_tid=14, entry_tid=21440], [object_number=87, tag_tid=14, entry_tid=21439], [object_number=86, tag_tid=14, entry_tid=21438]]], state=connected] 2024-04-24T00:49:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=21441], [object_number=84, tag_tid=14, entry_tid=21440], [object_number=87, tag_tid=14, entry_tid=21439], [object_number=86, tag_tid=14, entry_tid=21438]]], state=connected] 2024-04-24T00:49:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=21441' 2024-04-24T00:49:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=21441' 2024-04-24T00:49:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=21441' '!=' 'object_number=86, tag_tid=14, entry_tid=21298' 2024-04-24T00:49:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=21441' = 'object_number=86, tag_tid=14, entry_tid=21298' 2024-04-24T00:49:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=21441' 2024-04-24T00:49:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=21441' 2024-04-24T00:49:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=21441' 2024-04-24T00:49:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21441 2024-04-24T00:49:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 21441 2024-04-24T00:49:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=21578], [object_number=85, tag_tid=14, entry_tid=21577], [object_number=84, tag_tid=14, entry_tid=21576], [object_number=87, tag_tid=14, entry_tid=21575]]], state=connected] 2024-04-24T00:49:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=21578], [object_number=85, tag_tid=14, entry_tid=21577], [object_number=84, tag_tid=14, entry_tid=21576], [object_number=87, tag_tid=14, entry_tid=21575]]], state=connected] 2024-04-24T00:49:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=21578' 2024-04-24T00:49:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=21578' 2024-04-24T00:49:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=21578' '!=' 'object_number=85, tag_tid=14, entry_tid=21441' 2024-04-24T00:49:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=21578' = 'object_number=85, tag_tid=14, entry_tid=21441' 2024-04-24T00:49:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=21578' 2024-04-24T00:49:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=21578' 2024-04-24T00:49:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=21578' 2024-04-24T00:49:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21578 2024-04-24T00:49:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 21578 2024-04-24T00:49:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:01.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:01.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:01.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=21738], [object_number=85, tag_tid=14, entry_tid=21737], [object_number=84, tag_tid=14, entry_tid=21736], [object_number=87, tag_tid=14, entry_tid=21735]]], state=connected] 2024-04-24T00:49:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=21738], [object_number=85, tag_tid=14, entry_tid=21737], [object_number=84, tag_tid=14, entry_tid=21736], [object_number=87, tag_tid=14, entry_tid=21735]]], state=connected] 2024-04-24T00:49:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:02.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=21738' 2024-04-24T00:49:02.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=21738' 2024-04-24T00:49:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=21738' '!=' 'object_number=86, tag_tid=14, entry_tid=21578' 2024-04-24T00:49:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=21738' = 'object_number=86, tag_tid=14, entry_tid=21578' 2024-04-24T00:49:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=21738' 2024-04-24T00:49:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=21738' 2024-04-24T00:49:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=21738' 2024-04-24T00:49:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21738 2024-04-24T00:49:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 21738 2024-04-24T00:49:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=21905], [object_number=84, tag_tid=14, entry_tid=21904], [object_number=87, tag_tid=14, entry_tid=21903], [object_number=86, tag_tid=14, entry_tid=21902]]], state=connected] 2024-04-24T00:49:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=21905], [object_number=84, tag_tid=14, entry_tid=21904], [object_number=87, tag_tid=14, entry_tid=21903], [object_number=86, tag_tid=14, entry_tid=21902]]], state=connected] 2024-04-24T00:49:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=21905' 2024-04-24T00:49:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=21905' 2024-04-24T00:49:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=21905' '!=' 'object_number=86, tag_tid=14, entry_tid=21738' 2024-04-24T00:49:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=21905' = 'object_number=86, tag_tid=14, entry_tid=21738' 2024-04-24T00:49:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=21905' 2024-04-24T00:49:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=21905' 2024-04-24T00:49:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=21905' 2024-04-24T00:49:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=21905 2024-04-24T00:49:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 21905 2024-04-24T00:49:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=22085], [object_number=84, tag_tid=14, entry_tid=22084], [object_number=87, tag_tid=14, entry_tid=22083], [object_number=86, tag_tid=14, entry_tid=22082]]], state=connected] 2024-04-24T00:49:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:03.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=22085], [object_number=84, tag_tid=14, entry_tid=22084], [object_number=87, tag_tid=14, entry_tid=22083], [object_number=86, tag_tid=14, entry_tid=22082]]], state=connected] 2024-04-24T00:49:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=22085' 2024-04-24T00:49:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=22085' 2024-04-24T00:49:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=22085' '!=' 'object_number=85, tag_tid=14, entry_tid=21905' 2024-04-24T00:49:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=22085' = 'object_number=85, tag_tid=14, entry_tid=21905' 2024-04-24T00:49:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=22085' 2024-04-24T00:49:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=22085' 2024-04-24T00:49:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=22085' 2024-04-24T00:49:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22085 2024-04-24T00:49:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 22085 2024-04-24T00:49:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=22261], [object_number=84, tag_tid=14, entry_tid=22260], [object_number=87, tag_tid=14, entry_tid=22259], [object_number=86, tag_tid=14, entry_tid=22258]]], state=connected] 2024-04-24T00:49:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=22261], [object_number=84, tag_tid=14, entry_tid=22260], [object_number=87, tag_tid=14, entry_tid=22259], [object_number=86, tag_tid=14, entry_tid=22258]]], state=connected] 2024-04-24T00:49:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=22261' 2024-04-24T00:49:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=22261' 2024-04-24T00:49:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=22261' '!=' 'object_number=85, tag_tid=14, entry_tid=22085' 2024-04-24T00:49:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=22261' = 'object_number=85, tag_tid=14, entry_tid=22085' 2024-04-24T00:49:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=22261' 2024-04-24T00:49:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=22261' 2024-04-24T00:49:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=22261' 2024-04-24T00:49:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22261 2024-04-24T00:49:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 22261 2024-04-24T00:49:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:05.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:05.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=22435], [object_number=86, tag_tid=14, entry_tid=22434], [object_number=85, tag_tid=14, entry_tid=22433], [object_number=84, tag_tid=14, entry_tid=22432]]], state=connected] 2024-04-24T00:49:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=22435], [object_number=86, tag_tid=14, entry_tid=22434], [object_number=85, tag_tid=14, entry_tid=22433], [object_number=84, tag_tid=14, entry_tid=22432]]], state=connected] 2024-04-24T00:49:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=22435' 2024-04-24T00:49:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=22435' 2024-04-24T00:49:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=22435' '!=' 'object_number=85, tag_tid=14, entry_tid=22261' 2024-04-24T00:49:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=22435' = 'object_number=85, tag_tid=14, entry_tid=22261' 2024-04-24T00:49:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=22435' 2024-04-24T00:49:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=22435' 2024-04-24T00:49:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=22435' 2024-04-24T00:49:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22435 2024-04-24T00:49:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 22435 2024-04-24T00:49:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22602], [object_number=85, tag_tid=14, entry_tid=22601], [object_number=84, tag_tid=14, entry_tid=22600], [object_number=87, tag_tid=14, entry_tid=22599]]], state=connected] 2024-04-24T00:49:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22602], [object_number=85, tag_tid=14, entry_tid=22601], [object_number=84, tag_tid=14, entry_tid=22600], [object_number=87, tag_tid=14, entry_tid=22599]]], state=connected] 2024-04-24T00:49:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=22602' 2024-04-24T00:49:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=22602' 2024-04-24T00:49:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=22602' '!=' 'object_number=87, tag_tid=14, entry_tid=22435' 2024-04-24T00:49:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=22602' = 'object_number=87, tag_tid=14, entry_tid=22435' 2024-04-24T00:49:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=22602' 2024-04-24T00:49:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=22602' 2024-04-24T00:49:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=22602' 2024-04-24T00:49:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22602 2024-04-24T00:49:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 22602 2024-04-24T00:49:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=22768], [object_number=87, tag_tid=14, entry_tid=22767], [object_number=86, tag_tid=14, entry_tid=22766], [object_number=85, tag_tid=14, entry_tid=22765]]], state=connected] 2024-04-24T00:49:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=22768], [object_number=87, tag_tid=14, entry_tid=22767], [object_number=86, tag_tid=14, entry_tid=22766], [object_number=85, tag_tid=14, entry_tid=22765]]], state=connected] 2024-04-24T00:49:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=22768' 2024-04-24T00:49:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=22768' 2024-04-24T00:49:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=22768' '!=' 'object_number=86, tag_tid=14, entry_tid=22602' 2024-04-24T00:49:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=22768' = 'object_number=86, tag_tid=14, entry_tid=22602' 2024-04-24T00:49:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=22768' 2024-04-24T00:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=22768' 2024-04-24T00:49:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=22768' 2024-04-24T00:49:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22768 2024-04-24T00:49:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 22768 2024-04-24T00:49:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915], [object_number=86, tag_tid=14, entry_tid=22914], [object_number=85, tag_tid=14, entry_tid=22913]]], state=connected] 2024-04-24T00:49:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915], [object_number=86, tag_tid=14, entry_tid=22914], [object_number=85, tag_tid=14, entry_tid=22913]]], state=connected] 2024-04-24T00:49:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=22916' 2024-04-24T00:49:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=22916' 2024-04-24T00:49:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=22916' '!=' 'object_number=84, tag_tid=14, entry_tid=22768' 2024-04-24T00:49:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=22916' = 'object_number=84, tag_tid=14, entry_tid=22768' 2024-04-24T00:49:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=22916' 2024-04-24T00:49:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=22916' 2024-04-24T00:49:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=22916' 2024-04-24T00:49:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22916 2024-04-24T00:49:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 22916 2024-04-24T00:49:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22918], [object_number=85, tag_tid=14, entry_tid=22917], [object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22918], [object_number=85, tag_tid=14, entry_tid=22917], [object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=22918' '!=' 'object_number=84, tag_tid=14, entry_tid=22916' 2024-04-24T00:49:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:49:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=22918' = 'object_number=84, tag_tid=14, entry_tid=22916' 2024-04-24T00:49:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=15 2024-04-24T00:49:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:49:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=14 2024-04-24T00:49:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:49:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:49:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:49:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=22918 2024-04-24T00:49:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 15 = 14 -a 1 -le 22918 2024-04-24T00:49:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:49:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:49:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:09.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22918], [object_number=85, tag_tid=14, entry_tid=22917], [object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22918], [object_number=85, tag_tid=14, entry_tid=22917], [object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=22918' '!=' 'object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T00:49:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22918], [object_number=85, tag_tid=14, entry_tid=22917], [object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22918], [object_number=85, tag_tid=14, entry_tid=22917], [object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=22918' '!=' 'object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22918], [object_number=85, tag_tid=14, entry_tid=22917], [object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=22918], [object_number=85, tag_tid=14, entry_tid=22917], [object_number=84, tag_tid=14, entry_tid=22916], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=22918' '!=' 'object_number=86, tag_tid=14, entry_tid=22918' 2024-04-24T00:49:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:49:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T00:49:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:49:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:49:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:49:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:49:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:49:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:49:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:49:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:49:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:49:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:49:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:49:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:49:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:49:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:49:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:49:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:49:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:49:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:49:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:49:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:49:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:49:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:49:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:49:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:49:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:49:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:49:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:49:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:49:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:49:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:49:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:49:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:49:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:49:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:49:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:49:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:49:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:49:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:49:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:49:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:49:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:49:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:49:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:49:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:49:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:49:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:49:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:49:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:49:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:49:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:49:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:49:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:49:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:49:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:49:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:49:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:49:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:49:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:49:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:49:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:49:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:49:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:49:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:49:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:49:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:49:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:49:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:49:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:49:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 21 2024-04-24T00:49:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 21 2024-04-24T00:49:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:49:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=85, tag_tid=15, entry_tid=1], [object_number=84, tag_tid=15, entry_tid=0], [object_number=86, tag_tid=14, entry_tid=22918], [object_number=87, tag_tid=14, entry_tid=22915]]], state=connected] 2024-04-24T00:49:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:49:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=85, tag_tid=15, entry_tid=1' -a 'object_number=85, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=15, entry_tid=1' 2024-04-24T00:49:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-04-24T00:49:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5 2024-04-24T00:49:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:49:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:49:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:49:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-04-24T00:49:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:49:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:49:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:49:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 9 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:46:49 2024 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:49:04 2024 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:49:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap4-clean ']' 2024-04-24T00:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-24T00:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4-clean 2024-04-24T00:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:49:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:49:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:16.997 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:49:16.998+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:49:17.632 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:49:17.634+0000 7efd52cd0640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:49:17.632 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:49:17.634+0000 7efd534d1640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:49:17.635 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:49:17.634+0000 7efd51cce640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:49:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:49:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap5 2024-04-24T00:49:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5 2024-04-24T00:49:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:49:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:49:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:49:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:49:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:19.785 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:49:19.786+0000 7efd4ccc4640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:49:19.785 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:49:19.786+0000 7efd4fcca640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:49:19.786 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:49:19.786+0000 7efd4f4c9640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:49:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:49:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:49:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap5-clean 2024-04-24T00:49:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-04-24T00:49:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:49:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5-clean 2024-04-24T00:49:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:49:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:49:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:49:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:49:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:49:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=8 2024-04-24T00:49:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:49:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:49:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 49546240 bytes 2024-04-24T00:49:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 77815808 bytes 2024-04-24T00:49:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 118277120 bytes 2024-04-24T00:49:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 168219648 bytes 2024-04-24T00:49:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:49:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:49:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:49:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:49:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:49:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:49:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:49:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:49:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:49:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:49:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:49:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:49:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:49:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:49:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:49:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:49:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:49:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:49:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:49:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":1369553.57,"entries_behind_primary":1,"entries_per_second":86.17,"non_primary_position":{"entry_tid":1,"object_number":85,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":85,"tag_tid":16},"seconds_until_synced":0} 2024-04-24T00:49:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.3 on smithi134 2024-04-24T00:49:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:49:26 2024-04-24T00:49:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:49:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:49:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:49:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:49:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:49:26' 2024-04-24T00:49:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:49:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:49:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:49:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":1369553.57,"entries_behind_primary":1,"entries_per_second":86.17,"non_primary_position":{"entry_tid":1,"object_number":85,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":85,"tag_tid":16},"seconds_until_synced":0} 2024-04-24T00:49:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:49:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:49:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:49:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:49:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:49:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.3 on smithi134 2024-04-24T00:49:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:49:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:49:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:49:26' 2024-04-24T00:49:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:49:26' 2024-04-24T00:49:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:49:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:49:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:49:26' 2024-04-24T00:49:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:49:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:49:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:49:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142194274mirror.3smithi13419.0.0-2424-gb98aab6ctrueOK42594312mirror.2smithi13419.0.0-2424-gb98aab6cfalseOK44094412mirror.1smithi13419.0.0-2424-gb98aab6cfalseOK44534456mirror.0smithi13419.0.0-2424-gb98aab6cfalseOKtestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":1369553.57,"entries_behind_primary":1,"entries_per_second":86.17,"non_primary_position":{"entry_tid":1,"object_number":85,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":85,"tag_tid":16},"seconds_until_synced":0}42194274mirror.3smithi1342024-04-24 00:49:26cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:49:26' 2024-04-24T00:49:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:49:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:49:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:49:26' 2024-04-24T00:49:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:49:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:49:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:49:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:49:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:49:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:49:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6 2024-04-24T00:49:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap6 2024-04-24T00:49:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:49:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:49:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:49:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6 2024-04-24T00:49:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:49:33.906 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2920377803487183) > weight (0.25) 2024-04-24T00:49:33.906 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9944802374345355) > weight (0.25) 2024-04-24T00:49:33.906 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4468175748738167) > weight (0.25) 2024-04-24T00:49:33.907 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3422917954095974) > weight (0.25) 2024-04-24T00:49:33.907 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-04-24T00:50:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:50:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:50:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:50:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:50:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:50:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:50:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:50:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:50:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:50:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:50:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:50:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:50:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:50:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:50:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:50:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:50:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:50:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:50:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:50:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:50:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:50:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:50:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:50:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:50:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:50:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:50:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:50:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:50:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:50:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:50:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:50:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:50:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:50:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:50:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:50:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:50:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:50:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:50:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:50:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:50:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:50:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:50:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:50:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:50:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:50:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:50:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:50:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:50:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:50:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:50:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:50:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:50:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:50:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:50:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:50:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:50:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:50:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:50:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:50:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:50:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:50:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:50:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:50:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:50:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:50:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:50:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:50:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:50:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:50:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:50:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=6488], [object_number=95, tag_tid=17, entry_tid=6487], [object_number=94, tag_tid=17, entry_tid=6486], [object_number=93, tag_tid=17, entry_tid=6485]]], state=connected] 2024-04-24T00:50:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=6488], [object_number=95, tag_tid=17, entry_tid=6487], [object_number=94, tag_tid=17, entry_tid=6486], [object_number=93, tag_tid=17, entry_tid=6485]]], state=connected] 2024-04-24T00:50:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=6488' 2024-04-24T00:50:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=6488' 2024-04-24T00:50:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=6488' '!=' '' 2024-04-24T00:50:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=6488' = '' 2024-04-24T00:50:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=6488' 2024-04-24T00:50:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=6488' 2024-04-24T00:50:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=6488' 2024-04-24T00:50:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6488 2024-04-24T00:50:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 6488 2024-04-24T00:50:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=6635], [object_number=94, tag_tid=17, entry_tid=6634], [object_number=93, tag_tid=17, entry_tid=6633], [object_number=92, tag_tid=17, entry_tid=6632]]], state=connected] 2024-04-24T00:50:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=6635], [object_number=94, tag_tid=17, entry_tid=6634], [object_number=93, tag_tid=17, entry_tid=6633], [object_number=92, tag_tid=17, entry_tid=6632]]], state=connected] 2024-04-24T00:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=6635' 2024-04-24T00:50:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=6635' 2024-04-24T00:50:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=6635' '!=' 'object_number=92, tag_tid=17, entry_tid=6488' 2024-04-24T00:50:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=6635' = 'object_number=92, tag_tid=17, entry_tid=6488' 2024-04-24T00:50:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=6635' 2024-04-24T00:50:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=6635' 2024-04-24T00:50:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=6635' 2024-04-24T00:50:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6635 2024-04-24T00:50:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 6635 2024-04-24T00:50:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=6788], [object_number=95, tag_tid=17, entry_tid=6787], [object_number=94, tag_tid=17, entry_tid=6786], [object_number=93, tag_tid=17, entry_tid=6785]]], state=connected] 2024-04-24T00:50:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=6788], [object_number=95, tag_tid=17, entry_tid=6787], [object_number=94, tag_tid=17, entry_tid=6786], [object_number=93, tag_tid=17, entry_tid=6785]]], state=connected] 2024-04-24T00:50:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=6788' 2024-04-24T00:50:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=6788' 2024-04-24T00:50:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=6788' '!=' 'object_number=95, tag_tid=17, entry_tid=6635' 2024-04-24T00:50:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=6788' = 'object_number=95, tag_tid=17, entry_tid=6635' 2024-04-24T00:50:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=6788' 2024-04-24T00:50:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=6788' 2024-04-24T00:50:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=6788' 2024-04-24T00:50:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6788 2024-04-24T00:50:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 6788 2024-04-24T00:50:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=6939], [object_number=94, tag_tid=17, entry_tid=6938], [object_number=93, tag_tid=17, entry_tid=6937], [object_number=92, tag_tid=17, entry_tid=6936]]], state=connected] 2024-04-24T00:50:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=6939], [object_number=94, tag_tid=17, entry_tid=6938], [object_number=93, tag_tid=17, entry_tid=6937], [object_number=92, tag_tid=17, entry_tid=6936]]], state=connected] 2024-04-24T00:50:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=6939' 2024-04-24T00:50:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=6939' 2024-04-24T00:50:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=6939' '!=' 'object_number=92, tag_tid=17, entry_tid=6788' 2024-04-24T00:50:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=6939' = 'object_number=92, tag_tid=17, entry_tid=6788' 2024-04-24T00:50:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=6939' 2024-04-24T00:50:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=6939' 2024-04-24T00:50:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=6939' 2024-04-24T00:50:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6939 2024-04-24T00:50:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 6939 2024-04-24T00:50:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=7096], [object_number=95, tag_tid=17, entry_tid=7095], [object_number=94, tag_tid=17, entry_tid=7094], [object_number=93, tag_tid=17, entry_tid=7093]]], state=connected] 2024-04-24T00:50:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=7096], [object_number=95, tag_tid=17, entry_tid=7095], [object_number=94, tag_tid=17, entry_tid=7094], [object_number=93, tag_tid=17, entry_tid=7093]]], state=connected] 2024-04-24T00:50:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=7096' 2024-04-24T00:50:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=7096' 2024-04-24T00:50:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=7096' '!=' 'object_number=95, tag_tid=17, entry_tid=6939' 2024-04-24T00:50:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=7096' = 'object_number=95, tag_tid=17, entry_tid=6939' 2024-04-24T00:50:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=7096' 2024-04-24T00:50:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=7096' 2024-04-24T00:50:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=7096' 2024-04-24T00:50:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7096 2024-04-24T00:50:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 7096 2024-04-24T00:50:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=7251], [object_number=94, tag_tid=17, entry_tid=7250], [object_number=93, tag_tid=17, entry_tid=7249], [object_number=92, tag_tid=17, entry_tid=7248]]], state=connected] 2024-04-24T00:50:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=7251], [object_number=94, tag_tid=17, entry_tid=7250], [object_number=93, tag_tid=17, entry_tid=7249], [object_number=92, tag_tid=17, entry_tid=7248]]], state=connected] 2024-04-24T00:50:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=7251' 2024-04-24T00:50:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=7251' 2024-04-24T00:50:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=7251' '!=' 'object_number=92, tag_tid=17, entry_tid=7096' 2024-04-24T00:50:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=7251' = 'object_number=92, tag_tid=17, entry_tid=7096' 2024-04-24T00:50:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=7251' 2024-04-24T00:50:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=7251' 2024-04-24T00:50:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=7251' 2024-04-24T00:50:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7251 2024-04-24T00:50:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 7251 2024-04-24T00:50:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=7396], [object_number=95, tag_tid=17, entry_tid=7395], [object_number=94, tag_tid=17, entry_tid=7394], [object_number=93, tag_tid=17, entry_tid=7393]]], state=connected] 2024-04-24T00:50:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=7396], [object_number=95, tag_tid=17, entry_tid=7395], [object_number=94, tag_tid=17, entry_tid=7394], [object_number=93, tag_tid=17, entry_tid=7393]]], state=connected] 2024-04-24T00:50:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=7396' 2024-04-24T00:50:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=7396' 2024-04-24T00:50:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=7396' '!=' 'object_number=95, tag_tid=17, entry_tid=7251' 2024-04-24T00:50:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=7396' = 'object_number=95, tag_tid=17, entry_tid=7251' 2024-04-24T00:50:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=7396' 2024-04-24T00:50:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=7396' 2024-04-24T00:50:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=7396' 2024-04-24T00:50:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7396 2024-04-24T00:50:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 7396 2024-04-24T00:50:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=7538], [object_number=93, tag_tid=17, entry_tid=7537], [object_number=92, tag_tid=17, entry_tid=7536], [object_number=95, tag_tid=17, entry_tid=7535]]], state=connected] 2024-04-24T00:50:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=7538], [object_number=93, tag_tid=17, entry_tid=7537], [object_number=92, tag_tid=17, entry_tid=7536], [object_number=95, tag_tid=17, entry_tid=7535]]], state=connected] 2024-04-24T00:50:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=7538' 2024-04-24T00:50:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=7538' 2024-04-24T00:50:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=7538' '!=' 'object_number=92, tag_tid=17, entry_tid=7396' 2024-04-24T00:50:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=7538' = 'object_number=92, tag_tid=17, entry_tid=7396' 2024-04-24T00:50:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=7538' 2024-04-24T00:50:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=7538' 2024-04-24T00:50:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:09.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=7538' 2024-04-24T00:50:09.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:09.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7538 2024-04-24T00:50:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 7538 2024-04-24T00:50:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=7673], [object_number=92, tag_tid=17, entry_tid=7672], [object_number=95, tag_tid=17, entry_tid=7671], [object_number=94, tag_tid=17, entry_tid=7670]]], state=connected] 2024-04-24T00:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=7673], [object_number=92, tag_tid=17, entry_tid=7672], [object_number=95, tag_tid=17, entry_tid=7671], [object_number=94, tag_tid=17, entry_tid=7670]]], state=connected] 2024-04-24T00:50:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=7673' 2024-04-24T00:50:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=7673' 2024-04-24T00:50:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=7673' '!=' 'object_number=94, tag_tid=17, entry_tid=7538' 2024-04-24T00:50:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=7673' = 'object_number=94, tag_tid=17, entry_tid=7538' 2024-04-24T00:50:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=7673' 2024-04-24T00:50:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=7673' 2024-04-24T00:50:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=7673' 2024-04-24T00:50:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7673 2024-04-24T00:50:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 7673 2024-04-24T00:50:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=7860], [object_number=95, tag_tid=17, entry_tid=7859], [object_number=94, tag_tid=17, entry_tid=7858], [object_number=93, tag_tid=17, entry_tid=7857]]], state=connected] 2024-04-24T00:50:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=7860], [object_number=95, tag_tid=17, entry_tid=7859], [object_number=94, tag_tid=17, entry_tid=7858], [object_number=93, tag_tid=17, entry_tid=7857]]], state=connected] 2024-04-24T00:50:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=7860' 2024-04-24T00:50:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=7860' 2024-04-24T00:50:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=7860' '!=' 'object_number=93, tag_tid=17, entry_tid=7673' 2024-04-24T00:50:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=7860' = 'object_number=93, tag_tid=17, entry_tid=7673' 2024-04-24T00:50:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=7860' 2024-04-24T00:50:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=7860' 2024-04-24T00:50:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=7860' 2024-04-24T00:50:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7860 2024-04-24T00:50:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 7860 2024-04-24T00:50:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8025], [object_number=92, tag_tid=17, entry_tid=8024], [object_number=95, tag_tid=17, entry_tid=8023], [object_number=94, tag_tid=17, entry_tid=8022]]], state=connected] 2024-04-24T00:50:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8025], [object_number=92, tag_tid=17, entry_tid=8024], [object_number=95, tag_tid=17, entry_tid=8023], [object_number=94, tag_tid=17, entry_tid=8022]]], state=connected] 2024-04-24T00:50:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=8025' 2024-04-24T00:50:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=8025' 2024-04-24T00:50:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8025' '!=' 'object_number=92, tag_tid=17, entry_tid=7860' 2024-04-24T00:50:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8025' = 'object_number=92, tag_tid=17, entry_tid=7860' 2024-04-24T00:50:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=8025' 2024-04-24T00:50:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8025' 2024-04-24T00:50:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8025' 2024-04-24T00:50:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8025 2024-04-24T00:50:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 8025 2024-04-24T00:50:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8173], [object_number=92, tag_tid=17, entry_tid=8172], [object_number=95, tag_tid=17, entry_tid=8171], [object_number=94, tag_tid=17, entry_tid=8170]]], state=connected] 2024-04-24T00:50:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8173], [object_number=92, tag_tid=17, entry_tid=8172], [object_number=95, tag_tid=17, entry_tid=8171], [object_number=94, tag_tid=17, entry_tid=8170]]], state=connected] 2024-04-24T00:50:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=8173' 2024-04-24T00:50:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=8173' 2024-04-24T00:50:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8173' '!=' 'object_number=93, tag_tid=17, entry_tid=8025' 2024-04-24T00:50:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8173' = 'object_number=93, tag_tid=17, entry_tid=8025' 2024-04-24T00:50:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=8173' 2024-04-24T00:50:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8173' 2024-04-24T00:50:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8173' 2024-04-24T00:50:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8173 2024-04-24T00:50:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 8173 2024-04-24T00:50:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8337], [object_number=92, tag_tid=17, entry_tid=8336], [object_number=95, tag_tid=17, entry_tid=8335], [object_number=94, tag_tid=17, entry_tid=8334]]], state=connected] 2024-04-24T00:50:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8337], [object_number=92, tag_tid=17, entry_tid=8336], [object_number=95, tag_tid=17, entry_tid=8335], [object_number=94, tag_tid=17, entry_tid=8334]]], state=connected] 2024-04-24T00:50:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=8337' 2024-04-24T00:50:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=8337' 2024-04-24T00:50:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8337' '!=' 'object_number=93, tag_tid=17, entry_tid=8173' 2024-04-24T00:50:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8337' = 'object_number=93, tag_tid=17, entry_tid=8173' 2024-04-24T00:50:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=8337' 2024-04-24T00:50:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8337' 2024-04-24T00:50:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8337' 2024-04-24T00:50:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8337 2024-04-24T00:50:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 8337 2024-04-24T00:50:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8501], [object_number=92, tag_tid=17, entry_tid=8500], [object_number=95, tag_tid=17, entry_tid=8499], [object_number=94, tag_tid=17, entry_tid=8498]]], state=connected] 2024-04-24T00:50:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8501], [object_number=92, tag_tid=17, entry_tid=8500], [object_number=95, tag_tid=17, entry_tid=8499], [object_number=94, tag_tid=17, entry_tid=8498]]], state=connected] 2024-04-24T00:50:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=8501' 2024-04-24T00:50:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=8501' 2024-04-24T00:50:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8501' '!=' 'object_number=93, tag_tid=17, entry_tid=8337' 2024-04-24T00:50:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8501' = 'object_number=93, tag_tid=17, entry_tid=8337' 2024-04-24T00:50:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=8501' 2024-04-24T00:50:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8501' 2024-04-24T00:50:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8501' 2024-04-24T00:50:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:14.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8501 2024-04-24T00:50:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 8501 2024-04-24T00:50:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=8684], [object_number=95, tag_tid=17, entry_tid=8683], [object_number=94, tag_tid=17, entry_tid=8682], [object_number=93, tag_tid=17, entry_tid=8681]]], state=connected] 2024-04-24T00:50:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=8684], [object_number=95, tag_tid=17, entry_tid=8683], [object_number=94, tag_tid=17, entry_tid=8682], [object_number=93, tag_tid=17, entry_tid=8681]]], state=connected] 2024-04-24T00:50:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=8684' 2024-04-24T00:50:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=8684' 2024-04-24T00:50:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=8684' '!=' 'object_number=93, tag_tid=17, entry_tid=8501' 2024-04-24T00:50:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=8684' = 'object_number=93, tag_tid=17, entry_tid=8501' 2024-04-24T00:50:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=8684' 2024-04-24T00:50:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=8684' 2024-04-24T00:50:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=8684' 2024-04-24T00:50:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8684 2024-04-24T00:50:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 8684 2024-04-24T00:50:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8845], [object_number=92, tag_tid=17, entry_tid=8844], [object_number=95, tag_tid=17, entry_tid=8843], [object_number=94, tag_tid=17, entry_tid=8842]]], state=connected] 2024-04-24T00:50:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=8845], [object_number=92, tag_tid=17, entry_tid=8844], [object_number=95, tag_tid=17, entry_tid=8843], [object_number=94, tag_tid=17, entry_tid=8842]]], state=connected] 2024-04-24T00:50:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=8845' 2024-04-24T00:50:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=8845' 2024-04-24T00:50:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8845' '!=' 'object_number=92, tag_tid=17, entry_tid=8684' 2024-04-24T00:50:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=8845' = 'object_number=92, tag_tid=17, entry_tid=8684' 2024-04-24T00:50:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=8845' 2024-04-24T00:50:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8845' 2024-04-24T00:50:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=8845' 2024-04-24T00:50:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8845 2024-04-24T00:50:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 8845 2024-04-24T00:50:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=9003], [object_number=94, tag_tid=17, entry_tid=9002], [object_number=93, tag_tid=17, entry_tid=9001], [object_number=92, tag_tid=17, entry_tid=9000]]], state=connected] 2024-04-24T00:50:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=9003], [object_number=94, tag_tid=17, entry_tid=9002], [object_number=93, tag_tid=17, entry_tid=9001], [object_number=92, tag_tid=17, entry_tid=9000]]], state=connected] 2024-04-24T00:50:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=9003' 2024-04-24T00:50:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=9003' 2024-04-24T00:50:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=9003' '!=' 'object_number=93, tag_tid=17, entry_tid=8845' 2024-04-24T00:50:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=9003' = 'object_number=93, tag_tid=17, entry_tid=8845' 2024-04-24T00:50:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=9003' 2024-04-24T00:50:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=9003' 2024-04-24T00:50:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=9003' 2024-04-24T00:50:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9003 2024-04-24T00:50:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 9003 2024-04-24T00:50:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=9158], [object_number=93, tag_tid=17, entry_tid=9157], [object_number=92, tag_tid=17, entry_tid=9156], [object_number=95, tag_tid=17, entry_tid=9155]]], state=connected] 2024-04-24T00:50:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=9158], [object_number=93, tag_tid=17, entry_tid=9157], [object_number=92, tag_tid=17, entry_tid=9156], [object_number=95, tag_tid=17, entry_tid=9155]]], state=connected] 2024-04-24T00:50:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=9158' 2024-04-24T00:50:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=9158' 2024-04-24T00:50:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=9158' '!=' 'object_number=95, tag_tid=17, entry_tid=9003' 2024-04-24T00:50:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=9158' = 'object_number=95, tag_tid=17, entry_tid=9003' 2024-04-24T00:50:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=9158' 2024-04-24T00:50:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=9158' 2024-04-24T00:50:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=9158' 2024-04-24T00:50:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9158 2024-04-24T00:50:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 9158 2024-04-24T00:50:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=9319], [object_number=94, tag_tid=17, entry_tid=9318], [object_number=93, tag_tid=17, entry_tid=9317], [object_number=92, tag_tid=17, entry_tid=9316]]], state=connected] 2024-04-24T00:50:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=9319], [object_number=94, tag_tid=17, entry_tid=9318], [object_number=93, tag_tid=17, entry_tid=9317], [object_number=92, tag_tid=17, entry_tid=9316]]], state=connected] 2024-04-24T00:50:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=9319' 2024-04-24T00:50:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=9319' 2024-04-24T00:50:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=9319' '!=' 'object_number=94, tag_tid=17, entry_tid=9158' 2024-04-24T00:50:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=9319' = 'object_number=94, tag_tid=17, entry_tid=9158' 2024-04-24T00:50:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=9319' 2024-04-24T00:50:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=9319' 2024-04-24T00:50:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=9319' 2024-04-24T00:50:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9319 2024-04-24T00:50:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 9319 2024-04-24T00:50:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=9476], [object_number=95, tag_tid=17, entry_tid=9475], [object_number=94, tag_tid=17, entry_tid=9474], [object_number=93, tag_tid=17, entry_tid=9473]]], state=connected] 2024-04-24T00:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=9476], [object_number=95, tag_tid=17, entry_tid=9475], [object_number=94, tag_tid=17, entry_tid=9474], [object_number=93, tag_tid=17, entry_tid=9473]]], state=connected] 2024-04-24T00:50:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=9476' 2024-04-24T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=9476' 2024-04-24T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=9476' '!=' 'object_number=95, tag_tid=17, entry_tid=9319' 2024-04-24T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=9476' = 'object_number=95, tag_tid=17, entry_tid=9319' 2024-04-24T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=9476' 2024-04-24T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=9476' 2024-04-24T00:50:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=9476' 2024-04-24T00:50:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9476 2024-04-24T00:50:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 9476 2024-04-24T00:50:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=9629], [object_number=92, tag_tid=17, entry_tid=9628], [object_number=95, tag_tid=17, entry_tid=9627], [object_number=94, tag_tid=17, entry_tid=9626]]], state=connected] 2024-04-24T00:50:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=9629], [object_number=92, tag_tid=17, entry_tid=9628], [object_number=95, tag_tid=17, entry_tid=9627], [object_number=94, tag_tid=17, entry_tid=9626]]], state=connected] 2024-04-24T00:50:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=9629' 2024-04-24T00:50:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=9629' 2024-04-24T00:50:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=9629' '!=' 'object_number=92, tag_tid=17, entry_tid=9476' 2024-04-24T00:50:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=9629' = 'object_number=92, tag_tid=17, entry_tid=9476' 2024-04-24T00:50:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=9629' 2024-04-24T00:50:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=9629' 2024-04-24T00:50:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=9629' 2024-04-24T00:50:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9629 2024-04-24T00:50:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 9629 2024-04-24T00:50:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=9801], [object_number=92, tag_tid=17, entry_tid=9800], [object_number=95, tag_tid=17, entry_tid=9799], [object_number=94, tag_tid=17, entry_tid=9798]]], state=connected] 2024-04-24T00:50:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 23 2024-04-24T00:50:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=9801], [object_number=92, tag_tid=17, entry_tid=9800], [object_number=95, tag_tid=17, entry_tid=9799], [object_number=94, tag_tid=17, entry_tid=9798]]], state=connected] 2024-04-24T00:50:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=9801' 2024-04-24T00:50:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=9801' 2024-04-24T00:50:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=9801' '!=' 'object_number=93, tag_tid=17, entry_tid=9629' 2024-04-24T00:50:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=9801' = 'object_number=93, tag_tid=17, entry_tid=9629' 2024-04-24T00:50:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=9801' 2024-04-24T00:50:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=9801' 2024-04-24T00:50:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=9801' 2024-04-24T00:50:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9801 2024-04-24T00:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 9801 2024-04-24T00:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=9932], [object_number=99, tag_tid=17, entry_tid=9931], [object_number=98, tag_tid=17, entry_tid=9930], [object_number=97, tag_tid=17, entry_tid=9929]]], state=connected] 2024-04-24T00:50:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=9932], [object_number=99, tag_tid=17, entry_tid=9931], [object_number=98, tag_tid=17, entry_tid=9930], [object_number=97, tag_tid=17, entry_tid=9929]]], state=connected] 2024-04-24T00:50:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=96, tag_tid=17, entry_tid=9932' 2024-04-24T00:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=96, tag_tid=17, entry_tid=9932' 2024-04-24T00:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=9932' '!=' 'object_number=93, tag_tid=17, entry_tid=9801' 2024-04-24T00:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=9932' = 'object_number=93, tag_tid=17, entry_tid=9801' 2024-04-24T00:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=96, tag_tid=17, entry_tid=9932' 2024-04-24T00:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=9932' 2024-04-24T00:50:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=9932' 2024-04-24T00:50:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9932 2024-04-24T00:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 9932 2024-04-24T00:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:23.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:23.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:23.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=10084], [object_number=99, tag_tid=17, entry_tid=10083], [object_number=98, tag_tid=17, entry_tid=10082], [object_number=97, tag_tid=17, entry_tid=10081]]], state=connected] 2024-04-24T00:50:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=10084], [object_number=99, tag_tid=17, entry_tid=10083], [object_number=98, tag_tid=17, entry_tid=10082], [object_number=97, tag_tid=17, entry_tid=10081]]], state=connected] 2024-04-24T00:50:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=96, tag_tid=17, entry_tid=10084' 2024-04-24T00:50:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=96, tag_tid=17, entry_tid=10084' 2024-04-24T00:50:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=10084' '!=' 'object_number=96, tag_tid=17, entry_tid=9932' 2024-04-24T00:50:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=10084' = 'object_number=96, tag_tid=17, entry_tid=9932' 2024-04-24T00:50:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=96, tag_tid=17, entry_tid=10084' 2024-04-24T00:50:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=10084' 2024-04-24T00:50:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=10084' 2024-04-24T00:50:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10084 2024-04-24T00:50:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 10084 2024-04-24T00:50:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:24.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:24.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=10256], [object_number=99, tag_tid=17, entry_tid=10255], [object_number=98, tag_tid=17, entry_tid=10254], [object_number=97, tag_tid=17, entry_tid=10253]]], state=connected] 2024-04-24T00:50:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=10256], [object_number=99, tag_tid=17, entry_tid=10255], [object_number=98, tag_tid=17, entry_tid=10254], [object_number=97, tag_tid=17, entry_tid=10253]]], state=connected] 2024-04-24T00:50:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=96, tag_tid=17, entry_tid=10256' 2024-04-24T00:50:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=96, tag_tid=17, entry_tid=10256' 2024-04-24T00:50:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=10256' '!=' 'object_number=96, tag_tid=17, entry_tid=10084' 2024-04-24T00:50:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=10256' = 'object_number=96, tag_tid=17, entry_tid=10084' 2024-04-24T00:50:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=96, tag_tid=17, entry_tid=10256' 2024-04-24T00:50:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=10256' 2024-04-24T00:50:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=10256' 2024-04-24T00:50:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10256 2024-04-24T00:50:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 10256 2024-04-24T00:50:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=10439], [object_number=98, tag_tid=17, entry_tid=10438], [object_number=97, tag_tid=17, entry_tid=10437], [object_number=96, tag_tid=17, entry_tid=10436]]], state=connected] 2024-04-24T00:50:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=10439], [object_number=98, tag_tid=17, entry_tid=10438], [object_number=97, tag_tid=17, entry_tid=10437], [object_number=96, tag_tid=17, entry_tid=10436]]], state=connected] 2024-04-24T00:50:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=99, tag_tid=17, entry_tid=10439' 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=99, tag_tid=17, entry_tid=10439' 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=10439' '!=' 'object_number=96, tag_tid=17, entry_tid=10256' 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=10439' = 'object_number=96, tag_tid=17, entry_tid=10256' 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=99, tag_tid=17, entry_tid=10439' 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=10439' 2024-04-24T00:50:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=10439' 2024-04-24T00:50:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10439 2024-04-24T00:50:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 10439 2024-04-24T00:50:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=17, entry_tid=10593], [object_number=96, tag_tid=17, entry_tid=10592], [object_number=99, tag_tid=17, entry_tid=10591], [object_number=98, tag_tid=17, entry_tid=10590]]], state=connected] 2024-04-24T00:50:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=17, entry_tid=10593], [object_number=96, tag_tid=17, entry_tid=10592], [object_number=99, tag_tid=17, entry_tid=10591], [object_number=98, tag_tid=17, entry_tid=10590]]], state=connected] 2024-04-24T00:50:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=97, tag_tid=17, entry_tid=10593' 2024-04-24T00:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=97, tag_tid=17, entry_tid=10593' 2024-04-24T00:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=97, tag_tid=17, entry_tid=10593' '!=' 'object_number=99, tag_tid=17, entry_tid=10439' 2024-04-24T00:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=97, tag_tid=17, entry_tid=10593' = 'object_number=99, tag_tid=17, entry_tid=10439' 2024-04-24T00:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=97, tag_tid=17, entry_tid=10593' 2024-04-24T00:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=17, entry_tid=10593' 2024-04-24T00:50:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=17, entry_tid=10593' 2024-04-24T00:50:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10593 2024-04-24T00:50:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 10593 2024-04-24T00:50:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=17, entry_tid=10749], [object_number=96, tag_tid=17, entry_tid=10748], [object_number=99, tag_tid=17, entry_tid=10747], [object_number=98, tag_tid=17, entry_tid=10746]]], state=connected] 2024-04-24T00:50:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=17, entry_tid=10749], [object_number=96, tag_tid=17, entry_tid=10748], [object_number=99, tag_tid=17, entry_tid=10747], [object_number=98, tag_tid=17, entry_tid=10746]]], state=connected] 2024-04-24T00:50:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=97, tag_tid=17, entry_tid=10749' 2024-04-24T00:50:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=97, tag_tid=17, entry_tid=10749' 2024-04-24T00:50:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=97, tag_tid=17, entry_tid=10749' '!=' 'object_number=97, tag_tid=17, entry_tid=10593' 2024-04-24T00:50:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=97, tag_tid=17, entry_tid=10749' = 'object_number=97, tag_tid=17, entry_tid=10593' 2024-04-24T00:50:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=97, tag_tid=17, entry_tid=10749' 2024-04-24T00:50:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=17, entry_tid=10749' 2024-04-24T00:50:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=17, entry_tid=10749' 2024-04-24T00:50:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10749 2024-04-24T00:50:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 10749 2024-04-24T00:50:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=10912], [object_number=99, tag_tid=17, entry_tid=10911], [object_number=98, tag_tid=17, entry_tid=10910], [object_number=97, tag_tid=17, entry_tid=10909]]], state=connected] 2024-04-24T00:50:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=10912], [object_number=99, tag_tid=17, entry_tid=10911], [object_number=98, tag_tid=17, entry_tid=10910], [object_number=97, tag_tid=17, entry_tid=10909]]], state=connected] 2024-04-24T00:50:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=96, tag_tid=17, entry_tid=10912' 2024-04-24T00:50:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=96, tag_tid=17, entry_tid=10912' 2024-04-24T00:50:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=10912' '!=' 'object_number=97, tag_tid=17, entry_tid=10749' 2024-04-24T00:50:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=10912' = 'object_number=97, tag_tid=17, entry_tid=10749' 2024-04-24T00:50:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=96, tag_tid=17, entry_tid=10912' 2024-04-24T00:50:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=10912' 2024-04-24T00:50:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=10912' 2024-04-24T00:50:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10912 2024-04-24T00:50:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 10912 2024-04-24T00:50:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=11064], [object_number=99, tag_tid=17, entry_tid=11063], [object_number=98, tag_tid=17, entry_tid=11062], [object_number=97, tag_tid=17, entry_tid=11061]]], state=connected] 2024-04-24T00:50:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=11064], [object_number=99, tag_tid=17, entry_tid=11063], [object_number=98, tag_tid=17, entry_tid=11062], [object_number=97, tag_tid=17, entry_tid=11061]]], state=connected] 2024-04-24T00:50:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=96, tag_tid=17, entry_tid=11064' 2024-04-24T00:50:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=96, tag_tid=17, entry_tid=11064' 2024-04-24T00:50:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=11064' '!=' 'object_number=96, tag_tid=17, entry_tid=10912' 2024-04-24T00:50:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=11064' = 'object_number=96, tag_tid=17, entry_tid=10912' 2024-04-24T00:50:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=96, tag_tid=17, entry_tid=11064' 2024-04-24T00:50:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=11064' 2024-04-24T00:50:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=11064' 2024-04-24T00:50:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11064 2024-04-24T00:50:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 11064 2024-04-24T00:50:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=11212], [object_number=99, tag_tid=17, entry_tid=11211], [object_number=98, tag_tid=17, entry_tid=11210], [object_number=97, tag_tid=17, entry_tid=11209]]], state=connected] 2024-04-24T00:50:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=11212], [object_number=99, tag_tid=17, entry_tid=11211], [object_number=98, tag_tid=17, entry_tid=11210], [object_number=97, tag_tid=17, entry_tid=11209]]], state=connected] 2024-04-24T00:50:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=96, tag_tid=17, entry_tid=11212' 2024-04-24T00:50:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=96, tag_tid=17, entry_tid=11212' 2024-04-24T00:50:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=11212' '!=' 'object_number=96, tag_tid=17, entry_tid=11064' 2024-04-24T00:50:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=11212' = 'object_number=96, tag_tid=17, entry_tid=11064' 2024-04-24T00:50:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=96, tag_tid=17, entry_tid=11212' 2024-04-24T00:50:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=11212' 2024-04-24T00:50:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=11212' 2024-04-24T00:50:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11212 2024-04-24T00:50:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 11212 2024-04-24T00:50:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=11388], [object_number=99, tag_tid=17, entry_tid=11387], [object_number=98, tag_tid=17, entry_tid=11386], [object_number=97, tag_tid=17, entry_tid=11385]]], state=connected] 2024-04-24T00:50:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=11388], [object_number=99, tag_tid=17, entry_tid=11387], [object_number=98, tag_tid=17, entry_tid=11386], [object_number=97, tag_tid=17, entry_tid=11385]]], state=connected] 2024-04-24T00:50:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=96, tag_tid=17, entry_tid=11388' 2024-04-24T00:50:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=96, tag_tid=17, entry_tid=11388' 2024-04-24T00:50:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=11388' '!=' 'object_number=96, tag_tid=17, entry_tid=11212' 2024-04-24T00:50:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=11388' = 'object_number=96, tag_tid=17, entry_tid=11212' 2024-04-24T00:50:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=96, tag_tid=17, entry_tid=11388' 2024-04-24T00:50:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=11388' 2024-04-24T00:50:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=11388' 2024-04-24T00:50:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11388 2024-04-24T00:50:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 11388 2024-04-24T00:50:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=11539], [object_number=98, tag_tid=17, entry_tid=11538], [object_number=97, tag_tid=17, entry_tid=11537], [object_number=96, tag_tid=17, entry_tid=11536]]], state=connected] 2024-04-24T00:50:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=11539], [object_number=98, tag_tid=17, entry_tid=11538], [object_number=97, tag_tid=17, entry_tid=11537], [object_number=96, tag_tid=17, entry_tid=11536]]], state=connected] 2024-04-24T00:50:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=99, tag_tid=17, entry_tid=11539' 2024-04-24T00:50:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=99, tag_tid=17, entry_tid=11539' 2024-04-24T00:50:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=11539' '!=' 'object_number=96, tag_tid=17, entry_tid=11388' 2024-04-24T00:50:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=11539' = 'object_number=96, tag_tid=17, entry_tid=11388' 2024-04-24T00:50:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=99, tag_tid=17, entry_tid=11539' 2024-04-24T00:50:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=11539' 2024-04-24T00:50:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:31.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=11539' 2024-04-24T00:50:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11539 2024-04-24T00:50:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 11539 2024-04-24T00:50:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=11683], [object_number=98, tag_tid=17, entry_tid=11682], [object_number=97, tag_tid=17, entry_tid=11681], [object_number=96, tag_tid=17, entry_tid=11680]]], state=connected] 2024-04-24T00:50:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=11683], [object_number=98, tag_tid=17, entry_tid=11682], [object_number=97, tag_tid=17, entry_tid=11681], [object_number=96, tag_tid=17, entry_tid=11680]]], state=connected] 2024-04-24T00:50:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=99, tag_tid=17, entry_tid=11683' 2024-04-24T00:50:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=99, tag_tid=17, entry_tid=11683' 2024-04-24T00:50:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=11683' '!=' 'object_number=99, tag_tid=17, entry_tid=11539' 2024-04-24T00:50:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=11683' = 'object_number=99, tag_tid=17, entry_tid=11539' 2024-04-24T00:50:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=99, tag_tid=17, entry_tid=11683' 2024-04-24T00:50:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=11683' 2024-04-24T00:50:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=11683' 2024-04-24T00:50:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11683 2024-04-24T00:50:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 11683 2024-04-24T00:50:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=11835], [object_number=98, tag_tid=17, entry_tid=11834], [object_number=97, tag_tid=17, entry_tid=11833], [object_number=96, tag_tid=17, entry_tid=11832]]], state=connected] 2024-04-24T00:50:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=11835], [object_number=98, tag_tid=17, entry_tid=11834], [object_number=97, tag_tid=17, entry_tid=11833], [object_number=96, tag_tid=17, entry_tid=11832]]], state=connected] 2024-04-24T00:50:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=99, tag_tid=17, entry_tid=11835' 2024-04-24T00:50:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=99, tag_tid=17, entry_tid=11835' 2024-04-24T00:50:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=11835' '!=' 'object_number=99, tag_tid=17, entry_tid=11683' 2024-04-24T00:50:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=11835' = 'object_number=99, tag_tid=17, entry_tid=11683' 2024-04-24T00:50:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=99, tag_tid=17, entry_tid=11835' 2024-04-24T00:50:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=11835' 2024-04-24T00:50:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=11835' 2024-04-24T00:50:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11835 2024-04-24T00:50:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 11835 2024-04-24T00:50:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=12028], [object_number=99, tag_tid=17, entry_tid=12027], [object_number=98, tag_tid=17, entry_tid=12026], [object_number=97, tag_tid=17, entry_tid=12025]]], state=connected] 2024-04-24T00:50:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=96, tag_tid=17, entry_tid=12028], [object_number=99, tag_tid=17, entry_tid=12027], [object_number=98, tag_tid=17, entry_tid=12026], [object_number=97, tag_tid=17, entry_tid=12025]]], state=connected] 2024-04-24T00:50:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=96, tag_tid=17, entry_tid=12028' 2024-04-24T00:50:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=96, tag_tid=17, entry_tid=12028' 2024-04-24T00:50:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=12028' '!=' 'object_number=99, tag_tid=17, entry_tid=11835' 2024-04-24T00:50:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=96, tag_tid=17, entry_tid=12028' = 'object_number=99, tag_tid=17, entry_tid=11835' 2024-04-24T00:50:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=96, tag_tid=17, entry_tid=12028' 2024-04-24T00:50:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=12028' 2024-04-24T00:50:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=96, tag_tid=17, entry_tid=12028' 2024-04-24T00:50:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12028 2024-04-24T00:50:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 12028 2024-04-24T00:50:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=12178], [object_number=97, tag_tid=17, entry_tid=12177], [object_number=96, tag_tid=17, entry_tid=12176], [object_number=99, tag_tid=17, entry_tid=12175]]], state=connected] 2024-04-24T00:50:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=12178], [object_number=97, tag_tid=17, entry_tid=12177], [object_number=96, tag_tid=17, entry_tid=12176], [object_number=99, tag_tid=17, entry_tid=12175]]], state=connected] 2024-04-24T00:50:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=98, tag_tid=17, entry_tid=12178' 2024-04-24T00:50:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=98, tag_tid=17, entry_tid=12178' 2024-04-24T00:50:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=12178' '!=' 'object_number=96, tag_tid=17, entry_tid=12028' 2024-04-24T00:50:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=12178' = 'object_number=96, tag_tid=17, entry_tid=12028' 2024-04-24T00:50:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=98, tag_tid=17, entry_tid=12178' 2024-04-24T00:50:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=98, tag_tid=17, entry_tid=12178' 2024-04-24T00:50:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=98, tag_tid=17, entry_tid=12178' 2024-04-24T00:50:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12178 2024-04-24T00:50:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 12178 2024-04-24T00:50:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:35.800 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9018373297911098) > weight (0.25) 2024-04-24T00:50:35.800 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3507490151025995) > weight (0.25) 2024-04-24T00:50:35.800 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.298886446249654) > weight (0.25) 2024-04-24T00:50:35.800 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7652580376977739) > weight (0.25) 2024-04-24T00:50:35.800 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-04-24T00:50:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=12334], [object_number=97, tag_tid=17, entry_tid=12333], [object_number=96, tag_tid=17, entry_tid=12332], [object_number=99, tag_tid=17, entry_tid=12331]]], state=connected] 2024-04-24T00:50:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=12334], [object_number=97, tag_tid=17, entry_tid=12333], [object_number=96, tag_tid=17, entry_tid=12332], [object_number=99, tag_tid=17, entry_tid=12331]]], state=connected] 2024-04-24T00:50:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=98, tag_tid=17, entry_tid=12334' 2024-04-24T00:50:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=98, tag_tid=17, entry_tid=12334' 2024-04-24T00:50:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=12334' '!=' 'object_number=98, tag_tid=17, entry_tid=12178' 2024-04-24T00:50:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=12334' = 'object_number=98, tag_tid=17, entry_tid=12178' 2024-04-24T00:50:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=98, tag_tid=17, entry_tid=12334' 2024-04-24T00:50:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=98, tag_tid=17, entry_tid=12334' 2024-04-24T00:50:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=98, tag_tid=17, entry_tid=12334' 2024-04-24T00:50:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12334 2024-04-24T00:50:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 12334 2024-04-24T00:50:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=12498], [object_number=97, tag_tid=17, entry_tid=12497], [object_number=96, tag_tid=17, entry_tid=12496], [object_number=99, tag_tid=17, entry_tid=12495]]], state=connected] 2024-04-24T00:50:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=12498], [object_number=97, tag_tid=17, entry_tid=12497], [object_number=96, tag_tid=17, entry_tid=12496], [object_number=99, tag_tid=17, entry_tid=12495]]], state=connected] 2024-04-24T00:50:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=98, tag_tid=17, entry_tid=12498' 2024-04-24T00:50:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=98, tag_tid=17, entry_tid=12498' 2024-04-24T00:50:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=12498' '!=' 'object_number=98, tag_tid=17, entry_tid=12334' 2024-04-24T00:50:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=12498' = 'object_number=98, tag_tid=17, entry_tid=12334' 2024-04-24T00:50:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=98, tag_tid=17, entry_tid=12498' 2024-04-24T00:50:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=98, tag_tid=17, entry_tid=12498' 2024-04-24T00:50:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=98, tag_tid=17, entry_tid=12498' 2024-04-24T00:50:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12498 2024-04-24T00:50:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 12498 2024-04-24T00:50:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=17, entry_tid=12701], [object_number=96, tag_tid=17, entry_tid=12700], [object_number=99, tag_tid=17, entry_tid=12699], [object_number=98, tag_tid=17, entry_tid=12698]]], state=connected] 2024-04-24T00:50:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=17, entry_tid=12701], [object_number=96, tag_tid=17, entry_tid=12700], [object_number=99, tag_tid=17, entry_tid=12699], [object_number=98, tag_tid=17, entry_tid=12698]]], state=connected] 2024-04-24T00:50:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=97, tag_tid=17, entry_tid=12701' 2024-04-24T00:50:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=97, tag_tid=17, entry_tid=12701' 2024-04-24T00:50:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=97, tag_tid=17, entry_tid=12701' '!=' 'object_number=98, tag_tid=17, entry_tid=12498' 2024-04-24T00:50:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=97, tag_tid=17, entry_tid=12701' = 'object_number=98, tag_tid=17, entry_tid=12498' 2024-04-24T00:50:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=97, tag_tid=17, entry_tid=12701' 2024-04-24T00:50:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=17, entry_tid=12701' 2024-04-24T00:50:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=17, entry_tid=12701' 2024-04-24T00:50:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12701 2024-04-24T00:50:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 12701 2024-04-24T00:50:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=17, entry_tid=12865], [object_number=96, tag_tid=17, entry_tid=12864], [object_number=99, tag_tid=17, entry_tid=12863], [object_number=98, tag_tid=17, entry_tid=12862]]], state=connected] 2024-04-24T00:50:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=17, entry_tid=12865], [object_number=96, tag_tid=17, entry_tid=12864], [object_number=99, tag_tid=17, entry_tid=12863], [object_number=98, tag_tid=17, entry_tid=12862]]], state=connected] 2024-04-24T00:50:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=97, tag_tid=17, entry_tid=12865' 2024-04-24T00:50:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=97, tag_tid=17, entry_tid=12865' 2024-04-24T00:50:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=97, tag_tid=17, entry_tid=12865' '!=' 'object_number=97, tag_tid=17, entry_tid=12701' 2024-04-24T00:50:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=97, tag_tid=17, entry_tid=12865' = 'object_number=97, tag_tid=17, entry_tid=12701' 2024-04-24T00:50:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=97, tag_tid=17, entry_tid=12865' 2024-04-24T00:50:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=17, entry_tid=12865' 2024-04-24T00:50:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=17, entry_tid=12865' 2024-04-24T00:50:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12865 2024-04-24T00:50:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 12865 2024-04-24T00:50:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=13023], [object_number=98, tag_tid=17, entry_tid=13022], [object_number=97, tag_tid=17, entry_tid=13021], [object_number=96, tag_tid=17, entry_tid=13020]]], state=connected] 2024-04-24T00:50:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=99, tag_tid=17, entry_tid=13023], [object_number=98, tag_tid=17, entry_tid=13022], [object_number=97, tag_tid=17, entry_tid=13021], [object_number=96, tag_tid=17, entry_tid=13020]]], state=connected] 2024-04-24T00:50:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=99, tag_tid=17, entry_tid=13023' 2024-04-24T00:50:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=99, tag_tid=17, entry_tid=13023' 2024-04-24T00:50:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=13023' '!=' 'object_number=97, tag_tid=17, entry_tid=12865' 2024-04-24T00:50:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=99, tag_tid=17, entry_tid=13023' = 'object_number=97, tag_tid=17, entry_tid=12865' 2024-04-24T00:50:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=99, tag_tid=17, entry_tid=13023' 2024-04-24T00:50:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=13023' 2024-04-24T00:50:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=99, tag_tid=17, entry_tid=13023' 2024-04-24T00:50:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13023 2024-04-24T00:50:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 13023 2024-04-24T00:50:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=13034], [object_number=97, tag_tid=17, entry_tid=13033], [object_number=96, tag_tid=17, entry_tid=13032], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=13034], [object_number=97, tag_tid=17, entry_tid=13033], [object_number=96, tag_tid=17, entry_tid=13032], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=13034' '!=' 'object_number=99, tag_tid=17, entry_tid=13023' 2024-04-24T00:50:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:50:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=13034' = 'object_number=99, tag_tid=17, entry_tid=13023' 2024-04-24T00:50:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=18 2024-04-24T00:50:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:50:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=17 2024-04-24T00:50:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:50:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:50:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:50:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13034 2024-04-24T00:50:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 18 = 17 -a 1 -le 13034 2024-04-24T00:50:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:50:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:50:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=13034], [object_number=97, tag_tid=17, entry_tid=13033], [object_number=96, tag_tid=17, entry_tid=13032], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=13034], [object_number=97, tag_tid=17, entry_tid=13033], [object_number=96, tag_tid=17, entry_tid=13032], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=13034' '!=' 'object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T00:50:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=13034], [object_number=97, tag_tid=17, entry_tid=13033], [object_number=96, tag_tid=17, entry_tid=13032], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=13034], [object_number=97, tag_tid=17, entry_tid=13033], [object_number=96, tag_tid=17, entry_tid=13032], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=13034' '!=' 'object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T00:50:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=13034], [object_number=97, tag_tid=17, entry_tid=13033], [object_number=96, tag_tid=17, entry_tid=13032], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=98, tag_tid=17, entry_tid=13034], [object_number=97, tag_tid=17, entry_tid=13033], [object_number=96, tag_tid=17, entry_tid=13032], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=98, tag_tid=17, entry_tid=13034' '!=' 'object_number=98, tag_tid=17, entry_tid=13034' 2024-04-24T00:50:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:50:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:50:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:50:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:50:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:50:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:50:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:50:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:50:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:50:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:50:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:50:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:50:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:50:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:50:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:50:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:50:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:50:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:50:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:50:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:50:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:50:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:50:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:50:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:50:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:50:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:50:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:50:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:50:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:50:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:50:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:50:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:50:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:50:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:50:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:50:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:50:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:50:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:50:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:50:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:50:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:50:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:50:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:50:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:50:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:50:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:50:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:50:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:50:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:50:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:50:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:50:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:50:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:50:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:50:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:50:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:50:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:50:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:50:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:50:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:50:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:50:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:50:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:50:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:50:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:50:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:50:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:50:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:50:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:50:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:50:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:50:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:50:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:50:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:50:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:50:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:50:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:50:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:50:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 24 2024-04-24T00:50:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 24 2024-04-24T00:50:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:50:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=97, tag_tid=18, entry_tid=1], [object_number=96, tag_tid=18, entry_tid=0], [object_number=98, tag_tid=17, entry_tid=13034], [object_number=99, tag_tid=17, entry_tid=13031]]], state=connected] 2024-04-24T00:50:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:50:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=97, tag_tid=18, entry_tid=1' -a 'object_number=97, tag_tid=18, entry_tid=1' = 'object_number=97, tag_tid=18, entry_tid=1' 2024-04-24T00:50:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-04-24T00:50:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6 2024-04-24T00:50:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:50:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:50:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:50:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-04-24T00:50:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:50:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:50:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:50:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 11 2024-04-24T00:50:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:50:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:50:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:50:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:49:16 2024 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:50:05 2024 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:50:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:50:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap5-clean ']' 2024-04-24T00:50:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-24T00:50:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5-clean 2024-04-24T00:50:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:50:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:50:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:50:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:50:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:49.004 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:50:49.005+0000 7efd4f4c9640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:50:49.005 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:50:49.005+0000 7efd4ecc8640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:50:49.005 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:50:49.005+0000 7efd51cce640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:50:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:50:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap6 2024-04-24T00:50:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6 2024-04-24T00:50:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:50:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:50:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:50:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:50:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:50.872 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:50:50.873+0000 7efd51cce640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:50:50.873 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:50:50.873+0000 7efd4ecc8640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:50:51.088 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:50:51.089+0000 7fbc2ca22640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-04-24T00:50:51.088 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:50:51.089+0000 7fbc2aa1e640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 7 2024-04-24T00:50:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:50:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:50:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap6-clean 2024-04-24T00:50:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-04-24T00:50:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:50:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6-clean 2024-04-24T00:50:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:50:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:50:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:50:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:50:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:50:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T00:50:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:50:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:50:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 29870080 bytes 2024-04-24T00:50:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 46066688 bytes 2024-04-24T00:50:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 102895616 bytes 2024-04-24T00:50:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:50:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:50:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:50:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:50:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:50:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:50:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:50:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:50:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:50:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:50:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":1603862.9,"entries_behind_primary":1,"entries_per_second":100.83,"non_primary_position":{"entry_tid":1,"object_number":97,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":97,"tag_tid":19},"seconds_until_synced":0} 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.3 on smithi134 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:50:56 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:50:56' 2024-04-24T00:50:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:50:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:50:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:50:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":1603862.9,"entries_behind_primary":1,"entries_per_second":100.83,"non_primary_position":{"entry_tid":1,"object_number":97,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":97,"tag_tid":19},"seconds_until_synced":0} 2024-04-24T00:50:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:50:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:50:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:50:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:50:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:50:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.3 on smithi134 2024-04-24T00:50:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:50:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:50:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:50:56' 2024-04-24T00:50:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:50:56' 2024-04-24T00:50:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:50:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:50:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:50:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:50:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:50:56' 2024-04-24T00:50:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:50:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:50:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:51:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142194274mirror.3smithi13419.0.0-2424-gb98aab6ctrueOK42594312mirror.2smithi13419.0.0-2424-gb98aab6cfalseOK44094412mirror.1smithi13419.0.0-2424-gb98aab6cfalseOK44534456mirror.0smithi13419.0.0-2424-gb98aab6cfalseOKtestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":1603862.9,"entries_behind_primary":1,"entries_per_second":100.83,"non_primary_position":{"entry_tid":1,"object_number":97,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":97,"tag_tid":19},"seconds_until_synced":0}42194274mirror.3smithi1342024-04-24 00:50:56cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:50:56' 2024-04-24T00:51:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:51:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:51:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:50:56' 2024-04-24T00:51:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:51:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:51:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:51:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:51:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:51:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7 2024-04-24T00:51:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap7 2024-04-24T00:51:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:51:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7 2024-04-24T00:51:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:51:08.908 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T00:51:08.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T00:51:08.909 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-24T00:51:08.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T00:51:08.912+0000 7f1f79e22640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 65803) UID: 1000 2024-04-24T00:51:17.909 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T00:51:17.910 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:51:17.910 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T00:51:17.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T00:51:17.910 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T00:51:17.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T00:51:17.914 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-04-24T00:51:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:51:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:51:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:51:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:51:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:51:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:51:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:51:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:51:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:51:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:51:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:51:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:51:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:51:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:51:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:51:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:51:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:51:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:51:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:51:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:51:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:51:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:51:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:51:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:51:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:51:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:51:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:51:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:51:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:51:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:51:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:51:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:51:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:51:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:51:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:51:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:51:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:51:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:51:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:51:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:51:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:51:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:51:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:51:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:51:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:51:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:51:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:51:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:51:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:51:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:51:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:51:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:51:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:51:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:51:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:51:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:51:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:51:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:51:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:51:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:51:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:51:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:51:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:51:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:51:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:51:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:51:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=5949], [object_number=104, tag_tid=20, entry_tid=5948], [object_number=107, tag_tid=20, entry_tid=5947], [object_number=106, tag_tid=20, entry_tid=5946]]], state=connected] 2024-04-24T00:51:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=5949], [object_number=104, tag_tid=20, entry_tid=5948], [object_number=107, tag_tid=20, entry_tid=5947], [object_number=106, tag_tid=20, entry_tid=5946]]], state=connected] 2024-04-24T00:51:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=5949' 2024-04-24T00:51:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=5949' 2024-04-24T00:51:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=5949' '!=' '' 2024-04-24T00:51:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=5949' = '' 2024-04-24T00:51:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=5949' 2024-04-24T00:51:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=5949' 2024-04-24T00:51:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:31.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=5949' 2024-04-24T00:51:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5949 2024-04-24T00:51:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 5949 2024-04-24T00:51:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=6094], [object_number=105, tag_tid=20, entry_tid=6093], [object_number=104, tag_tid=20, entry_tid=6092], [object_number=107, tag_tid=20, entry_tid=6091]]], state=connected] 2024-04-24T00:51:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=6094], [object_number=105, tag_tid=20, entry_tid=6093], [object_number=104, tag_tid=20, entry_tid=6092], [object_number=107, tag_tid=20, entry_tid=6091]]], state=connected] 2024-04-24T00:51:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=6094' 2024-04-24T00:51:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=6094' 2024-04-24T00:51:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=6094' '!=' 'object_number=105, tag_tid=20, entry_tid=5949' 2024-04-24T00:51:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=6094' = 'object_number=105, tag_tid=20, entry_tid=5949' 2024-04-24T00:51:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=6094' 2024-04-24T00:51:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=6094' 2024-04-24T00:51:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=6094' 2024-04-24T00:51:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6094 2024-04-24T00:51:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 6094 2024-04-24T00:51:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=6263], [object_number=106, tag_tid=20, entry_tid=6262], [object_number=105, tag_tid=20, entry_tid=6261], [object_number=104, tag_tid=20, entry_tid=6260]]], state=connected] 2024-04-24T00:51:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=6263], [object_number=106, tag_tid=20, entry_tid=6262], [object_number=105, tag_tid=20, entry_tid=6261], [object_number=104, tag_tid=20, entry_tid=6260]]], state=connected] 2024-04-24T00:51:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=6263' 2024-04-24T00:51:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=6263' 2024-04-24T00:51:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=6263' '!=' 'object_number=106, tag_tid=20, entry_tid=6094' 2024-04-24T00:51:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=6263' = 'object_number=106, tag_tid=20, entry_tid=6094' 2024-04-24T00:51:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=6263' 2024-04-24T00:51:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=6263' 2024-04-24T00:51:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=6263' 2024-04-24T00:51:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6263 2024-04-24T00:51:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 6263 2024-04-24T00:51:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=6442], [object_number=105, tag_tid=20, entry_tid=6441], [object_number=104, tag_tid=20, entry_tid=6440], [object_number=107, tag_tid=20, entry_tid=6439]]], state=connected] 2024-04-24T00:51:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=6442], [object_number=105, tag_tid=20, entry_tid=6441], [object_number=104, tag_tid=20, entry_tid=6440], [object_number=107, tag_tid=20, entry_tid=6439]]], state=connected] 2024-04-24T00:51:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=6442' 2024-04-24T00:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=6442' 2024-04-24T00:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=6442' '!=' 'object_number=107, tag_tid=20, entry_tid=6263' 2024-04-24T00:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=6442' = 'object_number=107, tag_tid=20, entry_tid=6263' 2024-04-24T00:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=6442' 2024-04-24T00:51:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=6442' 2024-04-24T00:51:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=6442' 2024-04-24T00:51:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6442 2024-04-24T00:51:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 6442 2024-04-24T00:51:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=6606], [object_number=105, tag_tid=20, entry_tid=6605], [object_number=104, tag_tid=20, entry_tid=6604], [object_number=107, tag_tid=20, entry_tid=6603]]], state=connected] 2024-04-24T00:51:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:35.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=6606], [object_number=105, tag_tid=20, entry_tid=6605], [object_number=104, tag_tid=20, entry_tid=6604], [object_number=107, tag_tid=20, entry_tid=6603]]], state=connected] 2024-04-24T00:51:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=6606' 2024-04-24T00:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=6606' 2024-04-24T00:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=6606' '!=' 'object_number=106, tag_tid=20, entry_tid=6442' 2024-04-24T00:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=6606' = 'object_number=106, tag_tid=20, entry_tid=6442' 2024-04-24T00:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=6606' 2024-04-24T00:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=6606' 2024-04-24T00:51:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=6606' 2024-04-24T00:51:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6606 2024-04-24T00:51:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 6606 2024-04-24T00:51:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=6747], [object_number=106, tag_tid=20, entry_tid=6746], [object_number=105, tag_tid=20, entry_tid=6745], [object_number=104, tag_tid=20, entry_tid=6744]]], state=connected] 2024-04-24T00:51:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=6747], [object_number=106, tag_tid=20, entry_tid=6746], [object_number=105, tag_tid=20, entry_tid=6745], [object_number=104, tag_tid=20, entry_tid=6744]]], state=connected] 2024-04-24T00:51:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=6747' 2024-04-24T00:51:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=6747' 2024-04-24T00:51:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=6747' '!=' 'object_number=106, tag_tid=20, entry_tid=6606' 2024-04-24T00:51:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=6747' = 'object_number=106, tag_tid=20, entry_tid=6606' 2024-04-24T00:51:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=6747' 2024-04-24T00:51:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=6747' 2024-04-24T00:51:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=6747' 2024-04-24T00:51:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6747 2024-04-24T00:51:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 6747 2024-04-24T00:51:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=6892], [object_number=107, tag_tid=20, entry_tid=6891], [object_number=106, tag_tid=20, entry_tid=6890], [object_number=105, tag_tid=20, entry_tid=6889]]], state=connected] 2024-04-24T00:51:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=6892], [object_number=107, tag_tid=20, entry_tid=6891], [object_number=106, tag_tid=20, entry_tid=6890], [object_number=105, tag_tid=20, entry_tid=6889]]], state=connected] 2024-04-24T00:51:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=6892' 2024-04-24T00:51:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=6892' 2024-04-24T00:51:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=6892' '!=' 'object_number=107, tag_tid=20, entry_tid=6747' 2024-04-24T00:51:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=6892' = 'object_number=107, tag_tid=20, entry_tid=6747' 2024-04-24T00:51:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=6892' 2024-04-24T00:51:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=6892' 2024-04-24T00:51:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=6892' 2024-04-24T00:51:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6892 2024-04-24T00:51:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 6892 2024-04-24T00:51:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:37.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7044], [object_number=107, tag_tid=20, entry_tid=7043], [object_number=106, tag_tid=20, entry_tid=7042], [object_number=105, tag_tid=20, entry_tid=7041]]], state=connected] 2024-04-24T00:51:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7044], [object_number=107, tag_tid=20, entry_tid=7043], [object_number=106, tag_tid=20, entry_tid=7042], [object_number=105, tag_tid=20, entry_tid=7041]]], state=connected] 2024-04-24T00:51:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=7044' 2024-04-24T00:51:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=7044' 2024-04-24T00:51:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7044' '!=' 'object_number=104, tag_tid=20, entry_tid=6892' 2024-04-24T00:51:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7044' = 'object_number=104, tag_tid=20, entry_tid=6892' 2024-04-24T00:51:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=7044' 2024-04-24T00:51:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7044' 2024-04-24T00:51:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7044' 2024-04-24T00:51:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7044 2024-04-24T00:51:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 7044 2024-04-24T00:51:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=7193], [object_number=104, tag_tid=20, entry_tid=7192], [object_number=107, tag_tid=20, entry_tid=7191], [object_number=106, tag_tid=20, entry_tid=7190]]], state=connected] 2024-04-24T00:51:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=7193], [object_number=104, tag_tid=20, entry_tid=7192], [object_number=107, tag_tid=20, entry_tid=7191], [object_number=106, tag_tid=20, entry_tid=7190]]], state=connected] 2024-04-24T00:51:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=7193' 2024-04-24T00:51:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=7193' 2024-04-24T00:51:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=7193' '!=' 'object_number=104, tag_tid=20, entry_tid=7044' 2024-04-24T00:51:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=7193' = 'object_number=104, tag_tid=20, entry_tid=7044' 2024-04-24T00:51:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=7193' 2024-04-24T00:51:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=7193' 2024-04-24T00:51:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=7193' 2024-04-24T00:51:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7193 2024-04-24T00:51:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 7193 2024-04-24T00:51:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7336], [object_number=107, tag_tid=20, entry_tid=7335], [object_number=106, tag_tid=20, entry_tid=7334], [object_number=105, tag_tid=20, entry_tid=7333]]], state=connected] 2024-04-24T00:51:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7336], [object_number=107, tag_tid=20, entry_tid=7335], [object_number=106, tag_tid=20, entry_tid=7334], [object_number=105, tag_tid=20, entry_tid=7333]]], state=connected] 2024-04-24T00:51:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=7336' 2024-04-24T00:51:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=7336' 2024-04-24T00:51:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7336' '!=' 'object_number=105, tag_tid=20, entry_tid=7193' 2024-04-24T00:51:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7336' = 'object_number=105, tag_tid=20, entry_tid=7193' 2024-04-24T00:51:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=7336' 2024-04-24T00:51:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7336' 2024-04-24T00:51:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7336' 2024-04-24T00:51:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7336 2024-04-24T00:51:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 7336 2024-04-24T00:51:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=7474], [object_number=105, tag_tid=20, entry_tid=7473], [object_number=104, tag_tid=20, entry_tid=7472], [object_number=107, tag_tid=20, entry_tid=7471]]], state=connected] 2024-04-24T00:51:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=7474], [object_number=105, tag_tid=20, entry_tid=7473], [object_number=104, tag_tid=20, entry_tid=7472], [object_number=107, tag_tid=20, entry_tid=7471]]], state=connected] 2024-04-24T00:51:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=7474' 2024-04-24T00:51:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=7474' 2024-04-24T00:51:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=7474' '!=' 'object_number=104, tag_tid=20, entry_tid=7336' 2024-04-24T00:51:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=7474' = 'object_number=104, tag_tid=20, entry_tid=7336' 2024-04-24T00:51:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=7474' 2024-04-24T00:51:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=7474' 2024-04-24T00:51:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=7474' 2024-04-24T00:51:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7474 2024-04-24T00:51:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 7474 2024-04-24T00:51:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=7618], [object_number=105, tag_tid=20, entry_tid=7617], [object_number=104, tag_tid=20, entry_tid=7616], [object_number=107, tag_tid=20, entry_tid=7615]]], state=connected] 2024-04-24T00:51:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=7618], [object_number=105, tag_tid=20, entry_tid=7617], [object_number=104, tag_tid=20, entry_tid=7616], [object_number=107, tag_tid=20, entry_tid=7615]]], state=connected] 2024-04-24T00:51:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=7618' 2024-04-24T00:51:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=7618' 2024-04-24T00:51:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=7618' '!=' 'object_number=106, tag_tid=20, entry_tid=7474' 2024-04-24T00:51:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=7618' = 'object_number=106, tag_tid=20, entry_tid=7474' 2024-04-24T00:51:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=7618' 2024-04-24T00:51:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=7618' 2024-04-24T00:51:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=7618' 2024-04-24T00:51:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7618 2024-04-24T00:51:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 7618 2024-04-24T00:51:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=7777], [object_number=104, tag_tid=20, entry_tid=7776], [object_number=107, tag_tid=20, entry_tid=7775], [object_number=106, tag_tid=20, entry_tid=7774]]], state=connected] 2024-04-24T00:51:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=7777], [object_number=104, tag_tid=20, entry_tid=7776], [object_number=107, tag_tid=20, entry_tid=7775], [object_number=106, tag_tid=20, entry_tid=7774]]], state=connected] 2024-04-24T00:51:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=7777' 2024-04-24T00:51:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=7777' 2024-04-24T00:51:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=7777' '!=' 'object_number=106, tag_tid=20, entry_tid=7618' 2024-04-24T00:51:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=7777' = 'object_number=106, tag_tid=20, entry_tid=7618' 2024-04-24T00:51:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=7777' 2024-04-24T00:51:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=7777' 2024-04-24T00:51:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=7777' 2024-04-24T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7777 2024-04-24T00:51:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 7777 2024-04-24T00:51:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=7917], [object_number=104, tag_tid=20, entry_tid=7916], [object_number=107, tag_tid=20, entry_tid=7915], [object_number=106, tag_tid=20, entry_tid=7914]]], state=connected] 2024-04-24T00:51:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=7917], [object_number=104, tag_tid=20, entry_tid=7916], [object_number=107, tag_tid=20, entry_tid=7915], [object_number=106, tag_tid=20, entry_tid=7914]]], state=connected] 2024-04-24T00:51:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=7917' 2024-04-24T00:51:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=7917' 2024-04-24T00:51:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=7917' '!=' 'object_number=105, tag_tid=20, entry_tid=7777' 2024-04-24T00:51:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=7917' = 'object_number=105, tag_tid=20, entry_tid=7777' 2024-04-24T00:51:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=7917' 2024-04-24T00:51:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=7917' 2024-04-24T00:51:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=7917' 2024-04-24T00:51:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7917 2024-04-24T00:51:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 7917 2024-04-24T00:51:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8080], [object_number=107, tag_tid=20, entry_tid=8079], [object_number=106, tag_tid=20, entry_tid=8078], [object_number=105, tag_tid=20, entry_tid=8077]]], state=connected] 2024-04-24T00:51:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8080], [object_number=107, tag_tid=20, entry_tid=8079], [object_number=106, tag_tid=20, entry_tid=8078], [object_number=105, tag_tid=20, entry_tid=8077]]], state=connected] 2024-04-24T00:51:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=8080' 2024-04-24T00:51:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=8080' 2024-04-24T00:51:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8080' '!=' 'object_number=105, tag_tid=20, entry_tid=7917' 2024-04-24T00:51:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8080' = 'object_number=105, tag_tid=20, entry_tid=7917' 2024-04-24T00:51:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=8080' 2024-04-24T00:51:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8080' 2024-04-24T00:51:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8080' 2024-04-24T00:51:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8080 2024-04-24T00:51:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 8080 2024-04-24T00:51:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8249], [object_number=104, tag_tid=20, entry_tid=8248], [object_number=107, tag_tid=20, entry_tid=8247], [object_number=106, tag_tid=20, entry_tid=8246]]], state=connected] 2024-04-24T00:51:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8249], [object_number=104, tag_tid=20, entry_tid=8248], [object_number=107, tag_tid=20, entry_tid=8247], [object_number=106, tag_tid=20, entry_tid=8246]]], state=connected] 2024-04-24T00:51:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=8249' 2024-04-24T00:51:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=8249' 2024-04-24T00:51:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8249' '!=' 'object_number=104, tag_tid=20, entry_tid=8080' 2024-04-24T00:51:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8249' = 'object_number=104, tag_tid=20, entry_tid=8080' 2024-04-24T00:51:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=8249' 2024-04-24T00:51:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=8249' 2024-04-24T00:51:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=8249' 2024-04-24T00:51:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8249 2024-04-24T00:51:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 8249 2024-04-24T00:51:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:45.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:45.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:45.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8401], [object_number=104, tag_tid=20, entry_tid=8400], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8401], [object_number=104, tag_tid=20, entry_tid=8400], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8401' '!=' 'object_number=105, tag_tid=20, entry_tid=8249' 2024-04-24T00:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8401' = 'object_number=105, tag_tid=20, entry_tid=8249' 2024-04-24T00:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=21 2024-04-24T00:51:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:51:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=20 2024-04-24T00:51:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:51:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:51:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:51:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8401 2024-04-24T00:51:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 20 -a 1 -le 8401 2024-04-24T00:51:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:51:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:51:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8401], [object_number=104, tag_tid=20, entry_tid=8400], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8401], [object_number=104, tag_tid=20, entry_tid=8400], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8401' '!=' 'object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8401], [object_number=104, tag_tid=20, entry_tid=8400], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8401], [object_number=104, tag_tid=20, entry_tid=8400], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8401' '!=' 'object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T00:51:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8401], [object_number=104, tag_tid=20, entry_tid=8400], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8401], [object_number=104, tag_tid=20, entry_tid=8400], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8401' '!=' 'object_number=105, tag_tid=20, entry_tid=8401' 2024-04-24T00:51:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:51:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T00:51:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:51:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:51:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:51:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:51:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:51:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:51:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:51:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:51:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:51:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:51:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:51:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:51:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:51:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:51:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:51:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:51:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:51:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:51:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:51:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:51:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:51:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:51:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:51:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:51:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:51:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:51:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:51:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:51:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:51:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:51:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:51:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:51:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:51:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:51:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:51:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:51:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:51:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:51:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:51:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:51:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:51:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:51:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:51:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:51:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:51:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:51:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:51:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:51:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:51:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:51:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:51:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:51:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:51:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:51:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:51:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:51:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:51:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 26 2024-04-24T00:51:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 26 2024-04-24T00:51:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:51:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=105, tag_tid=21, entry_tid=1], [object_number=104, tag_tid=21, entry_tid=0], [object_number=107, tag_tid=20, entry_tid=8399], [object_number=106, tag_tid=20, entry_tid=8398]]], state=connected] 2024-04-24T00:51:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:51:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=105, tag_tid=21, entry_tid=1' -a 'object_number=105, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=21, entry_tid=1' 2024-04-24T00:51:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-04-24T00:51:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:51:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7 2024-04-24T00:51:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:51:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:51:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:51:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-04-24T00:51:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:51:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:51:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:51:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 13 2024-04-24T00:51:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:51:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:51:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:51:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:50:47 2024 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:51:06 2024 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:51:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:51:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap6-clean ']' 2024-04-24T00:51:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-24T00:51:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6-clean 2024-04-24T00:51:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:51:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:51:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:51:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:51:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:54.278 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:51:54.276+0000 7fbc2c221640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:51:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:51:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap7 2024-04-24T00:51:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7 2024-04-24T00:51:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:51:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:51:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:51:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:51:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:51:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:51:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:51:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap7-clean 2024-04-24T00:51:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-04-24T00:51:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:51:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7-clean 2024-04-24T00:51:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:51:57.765 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:51:57.767+0000 7f6308767640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 10 2024-04-24T00:51:57.765 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:51:57.767+0000 7f630275b640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-04-24T00:51:57.768 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:51:57.771+0000 7f630475f640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 17 2024-04-24T00:51:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:51:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:51:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:51:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:51:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:51:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:51:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T00:51:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:51:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:51:58.704 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:51:58.703+0000 7f6308767640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 18 2024-04-24T00:51:58.704 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:51:58.703+0000 7f6306763640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 22 2024-04-24T00:51:58.704 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:51:58.703+0000 7f630475f640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 22 2024-04-24T00:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 79044608 bytes 2024-04-24T00:52:02.892 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:52:02.891+0000 7fbdbe182640 -1 reset not still connected to 0x5622e1a31ba0 2024-04-24T00:52:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:52:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:52:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:52:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:52:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:52:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:52:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:52:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:52:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:52:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":2131230.63,"entries_behind_primary":0,"entries_per_second":132.83,"non_primary_position":{"entry_tid":1,"object_number":105,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":105,"tag_tid":21}} 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.3 on smithi134 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:51:56 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:51:56' 2024-04-24T00:52:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:52:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:52:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:52:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":2131230.63,"entries_behind_primary":0,"entries_per_second":132.83,"non_primary_position":{"entry_tid":1,"object_number":105,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":105,"tag_tid":21}} 2024-04-24T00:52:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:52:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:52:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:52:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:52:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:52:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.3 on smithi134 2024-04-24T00:52:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:52:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:52:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:51:56' 2024-04-24T00:52:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:51:56' 2024-04-24T00:52:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:52:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:51:56' 2024-04-24T00:52:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:52:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:52:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:52:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142194274mirror.3smithi13419.0.0-2424-gb98aab6ctrueOK42594312mirror.2smithi13419.0.0-2424-gb98aab6cfalseOK44094412mirror.1smithi13419.0.0-2424-gb98aab6cfalseOK44534456mirror.0smithi13419.0.0-2424-gb98aab6cfalseOKtestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":2131230.63,"entries_behind_primary":0,"entries_per_second":132.83,"non_primary_position":{"entry_tid":1,"object_number":105,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":105,"tag_tid":21}}42194274mirror.3smithi1342024-04-24 00:51:56cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:51:56' 2024-04-24T00:52:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:52:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:52:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:51:56' 2024-04-24T00:52:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:52:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:52:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:52:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:52:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:52:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8 2024-04-24T00:52:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap8 2024-04-24T00:52:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:52:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8 2024-04-24T00:52:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:52:20.801 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9944475446835628) > weight (0.25) 2024-04-24T00:52:20.801 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T00:52:20.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T00:52:20.802 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-24T00:52:20.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T00:52:20.802+0000 7f7d57d81640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 58600) UID: 1000 2024-04-24T00:52:26.802 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T00:52:26.802 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:52:26.803 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T00:52:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T00:52:26.803 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T00:52:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T00:52:26.806 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-04-24T00:52:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:52:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:52:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:52:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:52:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:52:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:52:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:52:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:52:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:52:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:52:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:52:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:52:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:52:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:52:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:52:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:52:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:52:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:52:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:52:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:52:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:52:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:52:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:52:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:52:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:52:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:52:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:52:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:52:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:52:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:52:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:52:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:52:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:52:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:52:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:52:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:52:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:52:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:52:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:52:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:52:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:52:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:52:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:52:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:52:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:52:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:52:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:52:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:52:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:52:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:52:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:52:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:52:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:52:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:52:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:52:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:52:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:52:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:52:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:52:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:52:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:52:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:52:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:52:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:52:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:52:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:52:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:52:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:52:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:52:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:52:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:52:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:52:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:52:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:52:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:52:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:52:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:52:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:52:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:52:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:52:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:52:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=114, tag_tid=23, entry_tid=6094], [object_number=113, tag_tid=23, entry_tid=6093], [object_number=112, tag_tid=23, entry_tid=6092], [object_number=115, tag_tid=23, entry_tid=6091]]], state=connected] 2024-04-24T00:52:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=114, tag_tid=23, entry_tid=6094], [object_number=113, tag_tid=23, entry_tid=6093], [object_number=112, tag_tid=23, entry_tid=6092], [object_number=115, tag_tid=23, entry_tid=6091]]], state=connected] 2024-04-24T00:52:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=114, tag_tid=23, entry_tid=6094' 2024-04-24T00:52:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=113, tag_tid=24, entry_tid=1' -a 'object_number=113, tag_tid=24, entry_tid=1' = 'object_number=114, tag_tid=23, entry_tid=6094' 2024-04-24T00:52:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=114, tag_tid=23, entry_tid=6094' '!=' '' 2024-04-24T00:52:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:52:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=114, tag_tid=23, entry_tid=6094' = '' 2024-04-24T00:52:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=114, tag_tid=23, entry_tid=6094' 2024-04-24T00:52:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=24 2024-04-24T00:52:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=114, tag_tid=23, entry_tid=6094' 2024-04-24T00:52:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=23 2024-04-24T00:52:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:52:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=114, tag_tid=23, entry_tid=6094' 2024-04-24T00:52:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6094 2024-04-24T00:52:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 24 = 23 -a 1 -le 6094 2024-04-24T00:52:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:52:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:52:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:52:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:52:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:52:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:52:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:52:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:52:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:52:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:52:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:52:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:52:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:52:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:52:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:52:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:52:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:52:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:52:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:52:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:52:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:52:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:52:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:52:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:52:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=115, tag_tid=23, entry_tid=6259], [object_number=114, tag_tid=23, entry_tid=6258], [object_number=113, tag_tid=23, entry_tid=6257], [object_number=112, tag_tid=23, entry_tid=6256]]], state=connected] 2024-04-24T00:52:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:52:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:52:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:52:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=115, tag_tid=23, entry_tid=6259], [object_number=114, tag_tid=23, entry_tid=6258], [object_number=113, tag_tid=23, entry_tid=6257], [object_number=112, tag_tid=23, entry_tid=6256]]], state=connected] 2024-04-24T00:52:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=115, tag_tid=23, entry_tid=6259' 2024-04-24T00:52:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=113, tag_tid=24, entry_tid=1' -a 'object_number=113, tag_tid=24, entry_tid=1' = 'object_number=115, tag_tid=23, entry_tid=6259' 2024-04-24T00:52:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=115, tag_tid=23, entry_tid=6259' '!=' 'object_number=114, tag_tid=23, entry_tid=6094' 2024-04-24T00:52:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:52:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=115, tag_tid=23, entry_tid=6259' = 'object_number=114, tag_tid=23, entry_tid=6094' 2024-04-24T00:52:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=115, tag_tid=23, entry_tid=6259' 2024-04-24T00:52:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=24 2024-04-24T00:52:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=115, tag_tid=23, entry_tid=6259' 2024-04-24T00:52:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=23 2024-04-24T00:52:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:52:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=115, tag_tid=23, entry_tid=6259' 2024-04-24T00:52:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6259 2024-04-24T00:52:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 24 = 23 -a 1 -le 6259 2024-04-24T00:52:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:52:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:52:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:52:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:52:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:52:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:52:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:52:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:52:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:52:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:52:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:52:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:52:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:52:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:52:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:52:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:52:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:52:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:52:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:52:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=114, tag_tid=23, entry_tid=6454], [object_number=113, tag_tid=23, entry_tid=6453], [object_number=112, tag_tid=23, entry_tid=6452], [object_number=115, tag_tid=23, entry_tid=6451]]], state=connected] 2024-04-24T00:52:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:52:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=114, tag_tid=23, entry_tid=6454], [object_number=113, tag_tid=23, entry_tid=6453], [object_number=112, tag_tid=23, entry_tid=6452], [object_number=115, tag_tid=23, entry_tid=6451]]], state=connected] 2024-04-24T00:52:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=114, tag_tid=23, entry_tid=6454' 2024-04-24T00:52:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=113, tag_tid=24, entry_tid=1' -a 'object_number=113, tag_tid=24, entry_tid=1' = 'object_number=114, tag_tid=23, entry_tid=6454' 2024-04-24T00:52:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=114, tag_tid=23, entry_tid=6454' '!=' 'object_number=115, tag_tid=23, entry_tid=6259' 2024-04-24T00:52:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:52:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=114, tag_tid=23, entry_tid=6454' = 'object_number=115, tag_tid=23, entry_tid=6259' 2024-04-24T00:52:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=114, tag_tid=23, entry_tid=6454' 2024-04-24T00:52:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=24 2024-04-24T00:52:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=114, tag_tid=23, entry_tid=6454' 2024-04-24T00:52:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=23 2024-04-24T00:52:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:52:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=114, tag_tid=23, entry_tid=6454' 2024-04-24T00:52:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6454 2024-04-24T00:52:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 24 = 23 -a 1 -le 6454 2024-04-24T00:52:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:52:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:52:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:52:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:52:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:52:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:52:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:52:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:52:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:52:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:52:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:52:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:52:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:52:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:52:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:52:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:52:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:52:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:52:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:52:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:52:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=113, tag_tid=23, entry_tid=6601], [object_number=112, tag_tid=23, entry_tid=6600], [object_number=115, tag_tid=23, entry_tid=6599], [object_number=114, tag_tid=23, entry_tid=6598]]], state=connected] 2024-04-24T00:52:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=113, tag_tid=23, entry_tid=6601], [object_number=112, tag_tid=23, entry_tid=6600], [object_number=115, tag_tid=23, entry_tid=6599], [object_number=114, tag_tid=23, entry_tid=6598]]], state=connected] 2024-04-24T00:52:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=113, tag_tid=23, entry_tid=6601' 2024-04-24T00:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=113, tag_tid=24, entry_tid=1' -a 'object_number=113, tag_tid=24, entry_tid=1' = 'object_number=113, tag_tid=23, entry_tid=6601' 2024-04-24T00:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=113, tag_tid=23, entry_tid=6601' '!=' 'object_number=114, tag_tid=23, entry_tid=6454' 2024-04-24T00:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=113, tag_tid=23, entry_tid=6601' = 'object_number=114, tag_tid=23, entry_tid=6454' 2024-04-24T00:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=113, tag_tid=23, entry_tid=6601' 2024-04-24T00:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=24 2024-04-24T00:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=23, entry_tid=6601' 2024-04-24T00:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=23 2024-04-24T00:52:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:52:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=23, entry_tid=6601' 2024-04-24T00:52:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6601 2024-04-24T00:52:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 24 = 23 -a 1 -le 6601 2024-04-24T00:52:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:52:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:52:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:52:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:52:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:52:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:52:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:52:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:52:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:52:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:52:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:52:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:52:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:52:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:52:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:52:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:52:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:52:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:52:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:52:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:52:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:52:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:52:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:52:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:52:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:52:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=113, tag_tid=23, entry_tid=6749], [object_number=112, tag_tid=23, entry_tid=6748], [object_number=115, tag_tid=23, entry_tid=6747], [object_number=114, tag_tid=23, entry_tid=6746]]], state=connected] 2024-04-24T00:52:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:52:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:52:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:52:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=113, tag_tid=23, entry_tid=6749], [object_number=112, tag_tid=23, entry_tid=6748], [object_number=115, tag_tid=23, entry_tid=6747], [object_number=114, tag_tid=23, entry_tid=6746]]], state=connected] 2024-04-24T00:52:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=113, tag_tid=23, entry_tid=6749' 2024-04-24T00:52:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=113, tag_tid=24, entry_tid=1' -a 'object_number=113, tag_tid=24, entry_tid=1' = 'object_number=113, tag_tid=23, entry_tid=6749' 2024-04-24T00:52:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=113, tag_tid=23, entry_tid=6749' '!=' 'object_number=113, tag_tid=23, entry_tid=6601' 2024-04-24T00:52:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:52:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=113, tag_tid=23, entry_tid=6749' = 'object_number=113, tag_tid=23, entry_tid=6601' 2024-04-24T00:52:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=113, tag_tid=23, entry_tid=6749' 2024-04-24T00:52:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=24 2024-04-24T00:52:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=23, entry_tid=6749' 2024-04-24T00:52:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=23 2024-04-24T00:52:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:52:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=23, entry_tid=6749' 2024-04-24T00:52:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6749 2024-04-24T00:52:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 24 = 23 -a 1 -le 6749 2024-04-24T00:52:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:52:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:52:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:52:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:52:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:52:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:52:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:52:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:52:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:52:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:52:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:52:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:52:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:52:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:52:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:52:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:52:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:52:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:52:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:52:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:52:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:52:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:52:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:52:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:52:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:52:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:52:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:52:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:52:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:52:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=112, tag_tid=23, entry_tid=6924], [object_number=115, tag_tid=23, entry_tid=6923], [object_number=114, tag_tid=23, entry_tid=6922], [object_number=113, tag_tid=23, entry_tid=6921]]], state=connected] 2024-04-24T00:52:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:52:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=112, tag_tid=23, entry_tid=6924], [object_number=115, tag_tid=23, entry_tid=6923], [object_number=114, tag_tid=23, entry_tid=6922], [object_number=113, tag_tid=23, entry_tid=6921]]], state=connected] 2024-04-24T00:52:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=112, tag_tid=23, entry_tid=6924' 2024-04-24T00:52:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=113, tag_tid=24, entry_tid=1' -a 'object_number=113, tag_tid=24, entry_tid=1' = 'object_number=112, tag_tid=23, entry_tid=6924' 2024-04-24T00:52:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=112, tag_tid=23, entry_tid=6924' '!=' 'object_number=113, tag_tid=23, entry_tid=6749' 2024-04-24T00:52:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:52:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=112, tag_tid=23, entry_tid=6924' = 'object_number=113, tag_tid=23, entry_tid=6749' 2024-04-24T00:52:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=112, tag_tid=23, entry_tid=6924' 2024-04-24T00:52:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=24 2024-04-24T00:52:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=112, tag_tid=23, entry_tid=6924' 2024-04-24T00:52:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=23 2024-04-24T00:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:52:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=112, tag_tid=23, entry_tid=6924' 2024-04-24T00:52:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6924 2024-04-24T00:52:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 24 = 23 -a 1 -le 6924 2024-04-24T00:52:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:52:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:52:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:52:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:52:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:52:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:52:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:52:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:52:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:52:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:52:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:52:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:52:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:52:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:52:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:52:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:52:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:52:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:52:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:52:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:52:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:52:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:52:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:52:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:52:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:52:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:52:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=115, tag_tid=23, entry_tid=7087], [object_number=114, tag_tid=23, entry_tid=7086], [object_number=113, tag_tid=23, entry_tid=7085], [object_number=112, tag_tid=23, entry_tid=7084]]], state=connected] 2024-04-24T00:52:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:52:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:52:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:52:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=115, tag_tid=23, entry_tid=7087], [object_number=114, tag_tid=23, entry_tid=7086], [object_number=113, tag_tid=23, entry_tid=7085], [object_number=112, tag_tid=23, entry_tid=7084]]], state=connected] 2024-04-24T00:52:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=115, tag_tid=23, entry_tid=7087' 2024-04-24T00:52:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=113, tag_tid=24, entry_tid=1' -a 'object_number=113, tag_tid=24, entry_tid=1' = 'object_number=115, tag_tid=23, entry_tid=7087' 2024-04-24T00:52:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=115, tag_tid=23, entry_tid=7087' '!=' 'object_number=112, tag_tid=23, entry_tid=6924' 2024-04-24T00:52:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:52:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=115, tag_tid=23, entry_tid=7087' = 'object_number=112, tag_tid=23, entry_tid=6924' 2024-04-24T00:52:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=115, tag_tid=23, entry_tid=7087' 2024-04-24T00:52:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=24 2024-04-24T00:52:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=115, tag_tid=23, entry_tid=7087' 2024-04-24T00:52:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=23 2024-04-24T00:52:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:52:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=115, tag_tid=23, entry_tid=7087' 2024-04-24T00:52:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7087 2024-04-24T00:52:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 24 = 23 -a 1 -le 7087 2024-04-24T00:52:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:52:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:52:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:52:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:52:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:52:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:52:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:52:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:52:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:52:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:52:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:52:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:52:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:52:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:52:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:52:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:52:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:52:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:52:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:52:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:52:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:52:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:52:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:52:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:52:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:52:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:52:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226], [object_number=113, tag_tid=23, entry_tid=7225], [object_number=112, tag_tid=23, entry_tid=7224]]], state=connected] 2024-04-24T00:52:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:52:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226], [object_number=113, tag_tid=23, entry_tid=7225], [object_number=112, tag_tid=23, entry_tid=7224]]], state=connected] 2024-04-24T00:52:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=115, tag_tid=23, entry_tid=7227' 2024-04-24T00:52:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=113, tag_tid=24, entry_tid=1' -a 'object_number=113, tag_tid=24, entry_tid=1' = 'object_number=115, tag_tid=23, entry_tid=7227' 2024-04-24T00:52:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=115, tag_tid=23, entry_tid=7227' '!=' 'object_number=115, tag_tid=23, entry_tid=7087' 2024-04-24T00:52:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:52:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=115, tag_tid=23, entry_tid=7227' = 'object_number=115, tag_tid=23, entry_tid=7087' 2024-04-24T00:52:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=115, tag_tid=23, entry_tid=7227' 2024-04-24T00:52:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=24 2024-04-24T00:52:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=115, tag_tid=23, entry_tid=7227' 2024-04-24T00:52:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:52:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=23 2024-04-24T00:52:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:52:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=115, tag_tid=23, entry_tid=7227' 2024-04-24T00:52:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:52:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:52:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7227 2024-04-24T00:52:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 24 = 23 -a 1 -le 7227 2024-04-24T00:52:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:52:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:52:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:52:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:52:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:52:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:52:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:52:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:52:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:52:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:52:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:52:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:52:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:52:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:52:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:52:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:52:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:52:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:52:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:52:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:52:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:52:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:52:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:52:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:52:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:52:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:52:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:52:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:52:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:52:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:52:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:52:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:52:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:52:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:52:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:52:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:52:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:52:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:52:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:52:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:52:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:52:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:52:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:52:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:52:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:52:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:52:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:52:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:52:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:52:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:52:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:52:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:52:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:52:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:52:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:52:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:52:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:52:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:52:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:52:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:52:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:52:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:52:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:52:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:52:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:52:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:52:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:52:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:52:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:52:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 28 2024-04-24T00:52:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 28 2024-04-24T00:52:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:52:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=113, tag_tid=24, entry_tid=1], [object_number=112, tag_tid=24, entry_tid=0], [object_number=115, tag_tid=23, entry_tid=7227], [object_number=114, tag_tid=23, entry_tid=7226]]], state=connected] 2024-04-24T00:52:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:52:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=113, tag_tid=24, entry_tid=1' -a 'object_number=113, tag_tid=24, entry_tid=1' = 'object_number=113, tag_tid=24, entry_tid=1' 2024-04-24T00:52:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-04-24T00:52:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8 2024-04-24T00:52:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:52:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:52:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:52:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-04-24T00:52:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:52:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:52:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:52:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 15 2024-04-24T00:52:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:52:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:52:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:51:53 2024 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:52:07 2024 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:52:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:52:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap7-clean ']' 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7-clean 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:52:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:52:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:49.127 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:52:49.130+0000 7efd4dcc6640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:52:49.330 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:52:49.334+0000 7efd4ecc8640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:52:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:52:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap8 2024-04-24T00:52:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8 2024-04-24T00:52:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:52:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:52:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:52:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:52:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:50.914 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.37264433579836254) > weight (0.25) 2024-04-24T00:52:50.915 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.48577303195119126) > weight (0.25) 2024-04-24T00:52:50.915 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5566343785981284) > weight (0.25) 2024-04-24T00:52:50.915 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.33795711706319775) > weight (0.25) 2024-04-24T00:52:50.915 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-04-24T00:52:51.303 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:52:51.306+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:52:51.308 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:52:51.310+0000 7efd514cd640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:52:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:52:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:52:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap8-clean 2024-04-24T00:52:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-04-24T00:52:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:52:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8-clean 2024-04-24T00:52:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:52:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:52:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:52:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:52:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T00:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:52:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 18637824 bytes 2024-04-24T00:52:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 62009344 bytes 2024-04-24T00:52:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 88677376 bytes 2024-04-24T00:52:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 128637952 bytes 2024-04-24T00:52:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:52:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:52:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:52:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:52:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:52:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:52:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:52:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:52:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:52:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:52:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:52:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:52:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:53:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:53:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:53:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:53:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:53:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:53:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:53:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:53:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:53:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:53:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:53:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":1766781.97,"entries_behind_primary":1,"entries_per_second":109.4,"non_primary_position":{"entry_tid":1,"object_number":113,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":113,"tag_tid":25},"seconds_until_synced":0} 2024-04-24T00:53:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.3 on smithi134 2024-04-24T00:53:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:52:56 2024-04-24T00:53:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:53:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:53:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:53:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:53:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:52:56' 2024-04-24T00:53:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:53:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:53:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:53:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":1766781.97,"entries_behind_primary":1,"entries_per_second":109.4,"non_primary_position":{"entry_tid":1,"object_number":113,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":113,"tag_tid":25},"seconds_until_synced":0} 2024-04-24T00:53:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:53:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:53:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:53:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:53:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:53:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.3 on smithi134 2024-04-24T00:53:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:53:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:53:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:52:56' 2024-04-24T00:53:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:52:56' 2024-04-24T00:53:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:53:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:52:56' 2024-04-24T00:53:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:53:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:53:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:53:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142194274mirror.3smithi13419.0.0-2424-gb98aab6ctrueOK42594312mirror.2smithi13419.0.0-2424-gb98aab6cfalseOK44534456mirror.0smithi13419.0.0-2424-gb98aab6cfalseOK45134516mirror.1smithi13419.0.0-2424-gb98aab6cfalseOKtestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":1766781.97,"entries_behind_primary":1,"entries_per_second":109.4,"non_primary_position":{"entry_tid":1,"object_number":113,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":113,"tag_tid":25},"seconds_until_synced":0}42194274mirror.3smithi1342024-04-24 00:52:56cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:52:56' 2024-04-24T00:53:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:53:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:53:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:52:56' 2024-04-24T00:53:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:53:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:53:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:53:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:53:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:53:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9 2024-04-24T00:53:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap9 2024-04-24T00:53:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:53:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9 2024-04-24T00:53:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:53:05.046 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:05.046+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa791520 2024-04-24T00:53:05.046 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:05.046+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db2616000 2024-04-24T00:53:05.046 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:05.046+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc0178680 2024-04-24T00:53:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:53:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:53:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:53:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:53:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:53:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:53:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:53:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:53:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:53:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:53:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:53:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:53:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:53:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:53:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:53:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:53:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:53:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:53:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:53:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:53:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:53:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:53:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:53:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:53:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:53:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:53:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:53:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:53:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:53:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:53:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:53:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:53:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:53:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:53:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:53:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:53:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:53:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:53:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:53:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:53:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:53:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:53:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:53:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:53:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:53:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:53:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:53:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:53:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:53:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:53:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:34.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:34.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:34.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:53:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:53:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:53:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:53:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:53:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:53:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:53:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:53:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:53:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:53:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:53:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=6223], [object_number=122, tag_tid=26, entry_tid=6222], [object_number=121, tag_tid=26, entry_tid=6221], [object_number=120, tag_tid=26, entry_tid=6220]]], state=connected] 2024-04-24T00:53:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=6223], [object_number=122, tag_tid=26, entry_tid=6222], [object_number=121, tag_tid=26, entry_tid=6221], [object_number=120, tag_tid=26, entry_tid=6220]]], state=connected] 2024-04-24T00:53:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=123, tag_tid=26, entry_tid=6223' 2024-04-24T00:53:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=123, tag_tid=26, entry_tid=6223' 2024-04-24T00:53:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=6223' '!=' '' 2024-04-24T00:53:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=6223' = '' 2024-04-24T00:53:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=123, tag_tid=26, entry_tid=6223' 2024-04-24T00:53:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=6223' 2024-04-24T00:53:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=6223' 2024-04-24T00:53:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6223 2024-04-24T00:53:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 6223 2024-04-24T00:53:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=120, tag_tid=26, entry_tid=6388], [object_number=123, tag_tid=26, entry_tid=6387], [object_number=122, tag_tid=26, entry_tid=6386], [object_number=121, tag_tid=26, entry_tid=6385]]], state=connected] 2024-04-24T00:53:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=120, tag_tid=26, entry_tid=6388], [object_number=123, tag_tid=26, entry_tid=6387], [object_number=122, tag_tid=26, entry_tid=6386], [object_number=121, tag_tid=26, entry_tid=6385]]], state=connected] 2024-04-24T00:53:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=120, tag_tid=26, entry_tid=6388' 2024-04-24T00:53:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=120, tag_tid=26, entry_tid=6388' 2024-04-24T00:53:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=120, tag_tid=26, entry_tid=6388' '!=' 'object_number=123, tag_tid=26, entry_tid=6223' 2024-04-24T00:53:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=120, tag_tid=26, entry_tid=6388' = 'object_number=123, tag_tid=26, entry_tid=6223' 2024-04-24T00:53:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=120, tag_tid=26, entry_tid=6388' 2024-04-24T00:53:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=120, tag_tid=26, entry_tid=6388' 2024-04-24T00:53:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=120, tag_tid=26, entry_tid=6388' 2024-04-24T00:53:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6388 2024-04-24T00:53:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 6388 2024-04-24T00:53:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=122, tag_tid=26, entry_tid=6542], [object_number=121, tag_tid=26, entry_tid=6541], [object_number=120, tag_tid=26, entry_tid=6540], [object_number=123, tag_tid=26, entry_tid=6539]]], state=connected] 2024-04-24T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=122, tag_tid=26, entry_tid=6542], [object_number=121, tag_tid=26, entry_tid=6541], [object_number=120, tag_tid=26, entry_tid=6540], [object_number=123, tag_tid=26, entry_tid=6539]]], state=connected] 2024-04-24T00:53:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=122, tag_tid=26, entry_tid=6542' 2024-04-24T00:53:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=122, tag_tid=26, entry_tid=6542' 2024-04-24T00:53:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=122, tag_tid=26, entry_tid=6542' '!=' 'object_number=120, tag_tid=26, entry_tid=6388' 2024-04-24T00:53:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=122, tag_tid=26, entry_tid=6542' = 'object_number=120, tag_tid=26, entry_tid=6388' 2024-04-24T00:53:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=122, tag_tid=26, entry_tid=6542' 2024-04-24T00:53:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=122, tag_tid=26, entry_tid=6542' 2024-04-24T00:53:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=122, tag_tid=26, entry_tid=6542' 2024-04-24T00:53:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6542 2024-04-24T00:53:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 6542 2024-04-24T00:53:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=26, entry_tid=6693], [object_number=120, tag_tid=26, entry_tid=6692], [object_number=123, tag_tid=26, entry_tid=6691], [object_number=122, tag_tid=26, entry_tid=6690]]], state=connected] 2024-04-24T00:53:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=26, entry_tid=6693], [object_number=120, tag_tid=26, entry_tid=6692], [object_number=123, tag_tid=26, entry_tid=6691], [object_number=122, tag_tid=26, entry_tid=6690]]], state=connected] 2024-04-24T00:53:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=121, tag_tid=26, entry_tid=6693' 2024-04-24T00:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=121, tag_tid=26, entry_tid=6693' 2024-04-24T00:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=121, tag_tid=26, entry_tid=6693' '!=' 'object_number=122, tag_tid=26, entry_tid=6542' 2024-04-24T00:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=121, tag_tid=26, entry_tid=6693' = 'object_number=122, tag_tid=26, entry_tid=6542' 2024-04-24T00:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=121, tag_tid=26, entry_tid=6693' 2024-04-24T00:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=26, entry_tid=6693' 2024-04-24T00:53:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=26, entry_tid=6693' 2024-04-24T00:53:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6693 2024-04-24T00:53:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 6693 2024-04-24T00:53:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=6843], [object_number=122, tag_tid=26, entry_tid=6842], [object_number=121, tag_tid=26, entry_tid=6841], [object_number=120, tag_tid=26, entry_tid=6840]]], state=connected] 2024-04-24T00:53:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=6843], [object_number=122, tag_tid=26, entry_tid=6842], [object_number=121, tag_tid=26, entry_tid=6841], [object_number=120, tag_tid=26, entry_tid=6840]]], state=connected] 2024-04-24T00:53:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=123, tag_tid=26, entry_tid=6843' 2024-04-24T00:53:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=123, tag_tid=26, entry_tid=6843' 2024-04-24T00:53:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=6843' '!=' 'object_number=121, tag_tid=26, entry_tid=6693' 2024-04-24T00:53:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=6843' = 'object_number=121, tag_tid=26, entry_tid=6693' 2024-04-24T00:53:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=123, tag_tid=26, entry_tid=6843' 2024-04-24T00:53:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=6843' 2024-04-24T00:53:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=6843' 2024-04-24T00:53:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6843 2024-04-24T00:53:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 6843 2024-04-24T00:53:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:39.278 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:39.277+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa099a00 2024-04-24T00:53:39.280 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:39.281+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa7900d0 2024-04-24T00:53:39.280 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:39.281+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa791520 2024-04-24T00:53:39.280 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:39.281+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db2616000 2024-04-24T00:53:39.281 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:39.281+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc0178680 2024-04-24T00:53:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=26, entry_tid=6997], [object_number=120, tag_tid=26, entry_tid=6996], [object_number=123, tag_tid=26, entry_tid=6995], [object_number=122, tag_tid=26, entry_tid=6994]]], state=connected] 2024-04-24T00:53:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=26, entry_tid=6997], [object_number=120, tag_tid=26, entry_tid=6996], [object_number=123, tag_tid=26, entry_tid=6995], [object_number=122, tag_tid=26, entry_tid=6994]]], state=connected] 2024-04-24T00:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=121, tag_tid=26, entry_tid=6997' 2024-04-24T00:53:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=121, tag_tid=26, entry_tid=6997' 2024-04-24T00:53:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=121, tag_tid=26, entry_tid=6997' '!=' 'object_number=123, tag_tid=26, entry_tid=6843' 2024-04-24T00:53:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=121, tag_tid=26, entry_tid=6997' = 'object_number=123, tag_tid=26, entry_tid=6843' 2024-04-24T00:53:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=121, tag_tid=26, entry_tid=6997' 2024-04-24T00:53:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=26, entry_tid=6997' 2024-04-24T00:53:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=26, entry_tid=6997' 2024-04-24T00:53:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6997 2024-04-24T00:53:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 6997 2024-04-24T00:53:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=120, tag_tid=26, entry_tid=7148], [object_number=123, tag_tid=26, entry_tid=7147], [object_number=122, tag_tid=26, entry_tid=7146], [object_number=121, tag_tid=26, entry_tid=7145]]], state=connected] 2024-04-24T00:53:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=120, tag_tid=26, entry_tid=7148], [object_number=123, tag_tid=26, entry_tid=7147], [object_number=122, tag_tid=26, entry_tid=7146], [object_number=121, tag_tid=26, entry_tid=7145]]], state=connected] 2024-04-24T00:53:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=120, tag_tid=26, entry_tid=7148' 2024-04-24T00:53:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=120, tag_tid=26, entry_tid=7148' 2024-04-24T00:53:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=120, tag_tid=26, entry_tid=7148' '!=' 'object_number=121, tag_tid=26, entry_tid=6997' 2024-04-24T00:53:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=120, tag_tid=26, entry_tid=7148' = 'object_number=121, tag_tid=26, entry_tid=6997' 2024-04-24T00:53:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=120, tag_tid=26, entry_tid=7148' 2024-04-24T00:53:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=120, tag_tid=26, entry_tid=7148' 2024-04-24T00:53:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=120, tag_tid=26, entry_tid=7148' 2024-04-24T00:53:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7148 2024-04-24T00:53:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 7148 2024-04-24T00:53:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=120, tag_tid=26, entry_tid=7308], [object_number=123, tag_tid=26, entry_tid=7307], [object_number=122, tag_tid=26, entry_tid=7306], [object_number=121, tag_tid=26, entry_tid=7305]]], state=connected] 2024-04-24T00:53:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=120, tag_tid=26, entry_tid=7308], [object_number=123, tag_tid=26, entry_tid=7307], [object_number=122, tag_tid=26, entry_tid=7306], [object_number=121, tag_tid=26, entry_tid=7305]]], state=connected] 2024-04-24T00:53:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=120, tag_tid=26, entry_tid=7308' 2024-04-24T00:53:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=120, tag_tid=26, entry_tid=7308' 2024-04-24T00:53:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=120, tag_tid=26, entry_tid=7308' '!=' 'object_number=120, tag_tid=26, entry_tid=7148' 2024-04-24T00:53:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=120, tag_tid=26, entry_tid=7308' = 'object_number=120, tag_tid=26, entry_tid=7148' 2024-04-24T00:53:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=120, tag_tid=26, entry_tid=7308' 2024-04-24T00:53:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=120, tag_tid=26, entry_tid=7308' 2024-04-24T00:53:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=120, tag_tid=26, entry_tid=7308' 2024-04-24T00:53:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7308 2024-04-24T00:53:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 7308 2024-04-24T00:53:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=122, tag_tid=26, entry_tid=7458], [object_number=121, tag_tid=26, entry_tid=7457], [object_number=120, tag_tid=26, entry_tid=7456], [object_number=123, tag_tid=26, entry_tid=7455]]], state=connected] 2024-04-24T00:53:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=122, tag_tid=26, entry_tid=7458], [object_number=121, tag_tid=26, entry_tid=7457], [object_number=120, tag_tid=26, entry_tid=7456], [object_number=123, tag_tid=26, entry_tid=7455]]], state=connected] 2024-04-24T00:53:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=122, tag_tid=26, entry_tid=7458' 2024-04-24T00:53:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=122, tag_tid=26, entry_tid=7458' 2024-04-24T00:53:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=122, tag_tid=26, entry_tid=7458' '!=' 'object_number=120, tag_tid=26, entry_tid=7308' 2024-04-24T00:53:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=122, tag_tid=26, entry_tid=7458' = 'object_number=120, tag_tid=26, entry_tid=7308' 2024-04-24T00:53:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=122, tag_tid=26, entry_tid=7458' 2024-04-24T00:53:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=122, tag_tid=26, entry_tid=7458' 2024-04-24T00:53:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=122, tag_tid=26, entry_tid=7458' 2024-04-24T00:53:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7458 2024-04-24T00:53:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 7458 2024-04-24T00:53:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=7603], [object_number=122, tag_tid=26, entry_tid=7602], [object_number=121, tag_tid=26, entry_tid=7601], [object_number=120, tag_tid=26, entry_tid=7600]]], state=connected] 2024-04-24T00:53:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=7603], [object_number=122, tag_tid=26, entry_tid=7602], [object_number=121, tag_tid=26, entry_tid=7601], [object_number=120, tag_tid=26, entry_tid=7600]]], state=connected] 2024-04-24T00:53:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=123, tag_tid=26, entry_tid=7603' 2024-04-24T00:53:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=123, tag_tid=26, entry_tid=7603' 2024-04-24T00:53:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=7603' '!=' 'object_number=122, tag_tid=26, entry_tid=7458' 2024-04-24T00:53:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=7603' = 'object_number=122, tag_tid=26, entry_tid=7458' 2024-04-24T00:53:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=123, tag_tid=26, entry_tid=7603' 2024-04-24T00:53:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=7603' 2024-04-24T00:53:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=7603' 2024-04-24T00:53:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7603 2024-04-24T00:53:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 7603 2024-04-24T00:53:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=26, entry_tid=7773], [object_number=120, tag_tid=26, entry_tid=7772], [object_number=123, tag_tid=26, entry_tid=7771], [object_number=122, tag_tid=26, entry_tid=7770]]], state=connected] 2024-04-24T00:53:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=26, entry_tid=7773], [object_number=120, tag_tid=26, entry_tid=7772], [object_number=123, tag_tid=26, entry_tid=7771], [object_number=122, tag_tid=26, entry_tid=7770]]], state=connected] 2024-04-24T00:53:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=121, tag_tid=26, entry_tid=7773' 2024-04-24T00:53:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=121, tag_tid=26, entry_tid=7773' 2024-04-24T00:53:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=121, tag_tid=26, entry_tid=7773' '!=' 'object_number=123, tag_tid=26, entry_tid=7603' 2024-04-24T00:53:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=121, tag_tid=26, entry_tid=7773' = 'object_number=123, tag_tid=26, entry_tid=7603' 2024-04-24T00:53:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=121, tag_tid=26, entry_tid=7773' 2024-04-24T00:53:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=26, entry_tid=7773' 2024-04-24T00:53:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=26, entry_tid=7773' 2024-04-24T00:53:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7773 2024-04-24T00:53:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 7773 2024-04-24T00:53:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=7939], [object_number=122, tag_tid=26, entry_tid=7938], [object_number=121, tag_tid=26, entry_tid=7937], [object_number=120, tag_tid=26, entry_tid=7936]]], state=connected] 2024-04-24T00:53:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=7939], [object_number=122, tag_tid=26, entry_tid=7938], [object_number=121, tag_tid=26, entry_tid=7937], [object_number=120, tag_tid=26, entry_tid=7936]]], state=connected] 2024-04-24T00:53:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=123, tag_tid=26, entry_tid=7939' 2024-04-24T00:53:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=123, tag_tid=26, entry_tid=7939' 2024-04-24T00:53:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=7939' '!=' 'object_number=121, tag_tid=26, entry_tid=7773' 2024-04-24T00:53:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=7939' = 'object_number=121, tag_tid=26, entry_tid=7773' 2024-04-24T00:53:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=123, tag_tid=26, entry_tid=7939' 2024-04-24T00:53:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=7939' 2024-04-24T00:53:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=7939' 2024-04-24T00:53:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7939 2024-04-24T00:53:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 7939 2024-04-24T00:53:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=8087], [object_number=122, tag_tid=26, entry_tid=8086], [object_number=121, tag_tid=26, entry_tid=8085], [object_number=120, tag_tid=26, entry_tid=8084]]], state=connected] 2024-04-24T00:53:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=123, tag_tid=26, entry_tid=8087], [object_number=122, tag_tid=26, entry_tid=8086], [object_number=121, tag_tid=26, entry_tid=8085], [object_number=120, tag_tid=26, entry_tid=8084]]], state=connected] 2024-04-24T00:53:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=123, tag_tid=26, entry_tid=8087' 2024-04-24T00:53:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=123, tag_tid=26, entry_tid=8087' 2024-04-24T00:53:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=8087' '!=' 'object_number=123, tag_tid=26, entry_tid=7939' 2024-04-24T00:53:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=123, tag_tid=26, entry_tid=8087' = 'object_number=123, tag_tid=26, entry_tid=7939' 2024-04-24T00:53:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=123, tag_tid=26, entry_tid=8087' 2024-04-24T00:53:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=8087' 2024-04-24T00:53:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=123, tag_tid=26, entry_tid=8087' 2024-04-24T00:53:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8087 2024-04-24T00:53:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 8087 2024-04-24T00:53:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=26, entry_tid=8225], [object_number=120, tag_tid=26, entry_tid=8224], [object_number=123, tag_tid=26, entry_tid=8223], [object_number=122, tag_tid=26, entry_tid=8222]]], state=connected] 2024-04-24T00:53:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=26, entry_tid=8225], [object_number=120, tag_tid=26, entry_tid=8224], [object_number=123, tag_tid=26, entry_tid=8223], [object_number=122, tag_tid=26, entry_tid=8222]]], state=connected] 2024-04-24T00:53:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=121, tag_tid=26, entry_tid=8225' 2024-04-24T00:53:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=121, tag_tid=26, entry_tid=8225' 2024-04-24T00:53:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=121, tag_tid=26, entry_tid=8225' '!=' 'object_number=123, tag_tid=26, entry_tid=8087' 2024-04-24T00:53:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=121, tag_tid=26, entry_tid=8225' = 'object_number=123, tag_tid=26, entry_tid=8087' 2024-04-24T00:53:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=121, tag_tid=26, entry_tid=8225' 2024-04-24T00:53:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=26, entry_tid=8225' 2024-04-24T00:53:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=26, entry_tid=8225' 2024-04-24T00:53:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8225 2024-04-24T00:53:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 8225 2024-04-24T00:53:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=120, tag_tid=26, entry_tid=8368], [object_number=123, tag_tid=26, entry_tid=8367], [object_number=122, tag_tid=26, entry_tid=8366], [object_number=121, tag_tid=26, entry_tid=8365]]], state=connected] 2024-04-24T00:53:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=120, tag_tid=26, entry_tid=8368], [object_number=123, tag_tid=26, entry_tid=8367], [object_number=122, tag_tid=26, entry_tid=8366], [object_number=121, tag_tid=26, entry_tid=8365]]], state=connected] 2024-04-24T00:53:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=120, tag_tid=26, entry_tid=8368' 2024-04-24T00:53:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=120, tag_tid=26, entry_tid=8368' 2024-04-24T00:53:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=120, tag_tid=26, entry_tid=8368' '!=' 'object_number=121, tag_tid=26, entry_tid=8225' 2024-04-24T00:53:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:53:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=120, tag_tid=26, entry_tid=8368' = 'object_number=121, tag_tid=26, entry_tid=8225' 2024-04-24T00:53:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=120, tag_tid=26, entry_tid=8368' 2024-04-24T00:53:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=27 2024-04-24T00:53:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=120, tag_tid=26, entry_tid=8368' 2024-04-24T00:53:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:53:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=26 2024-04-24T00:53:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:53:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=120, tag_tid=26, entry_tid=8368' 2024-04-24T00:53:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:53:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:53:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8368 2024-04-24T00:53:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 27 = 26 -a 1 -le 8368 2024-04-24T00:53:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:53:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:53:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:53:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:53:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:53:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:53:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:53:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:53:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:53:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:53:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:53:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:53:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:53:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:53:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:53:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:53:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:53:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:53:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:53:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:53:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:53:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:53:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:53:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:53:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:53:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:53:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:53:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:53:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:53:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:53:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:53:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:53:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:53:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:53:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:53:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:53:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:53:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:53:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:53:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:53:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:53:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:53:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:53:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:53:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:53:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:53:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:53:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:53:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:53:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:53:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:53:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:53:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:53:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:53:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:53:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:53:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:53:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:53:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:53:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:53:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:53:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:53:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:53:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:53:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:53:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:53:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:53:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:53:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:53:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:53:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:53:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:53:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:53:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:53:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 30 2024-04-24T00:53:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 30 2024-04-24T00:53:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:53:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=121, tag_tid=27, entry_tid=1], [object_number=120, tag_tid=27, entry_tid=0], [object_number=123, tag_tid=26, entry_tid=8383], [object_number=122, tag_tid=26, entry_tid=8382]]], state=connected] 2024-04-24T00:53:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:53:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=121, tag_tid=27, entry_tid=1' -a 'object_number=121, tag_tid=27, entry_tid=1' = 'object_number=121, tag_tid=27, entry_tid=1' 2024-04-24T00:53:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-04-24T00:53:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:53:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9 2024-04-24T00:53:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:53:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:53:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:53:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-04-24T00:53:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:53:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:53:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:53:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 17 2024-04-24T00:53:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:53:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:53:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:53:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:51:53 2024 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:53:08 2024 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:53:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:53:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap8-clean ']' 2024-04-24T00:53:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-24T00:53:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8-clean 2024-04-24T00:53:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:53:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:53:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:53:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:53:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:50.722 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:50.724+0000 7efd4dcc6640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:53:50.723 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:50.724+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:53:50.724 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:50.724+0000 7efd534d1640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:53:50.724 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:50.724+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:53:50.920 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:50.920+0000 7efd4dcc6640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:53:51.011 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:51.012+0000 7fbc2ba20640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:53:51.117 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:51.116+0000 7fbc28a1a640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:53:51.119 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:51.120+0000 7fbc2ea26640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:53:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:53:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap9 2024-04-24T00:53:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9 2024-04-24T00:53:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:53:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:53:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:53:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:53:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:52.879 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:52.880+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:53:53.099 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:53.100+0000 7efd4d4c5640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:53:53.483 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:53.484+0000 7fbc2ea26640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:53:53.483 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:53.484+0000 7fbc2ca22640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:53:53.485 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:53.484+0000 7fbc2f227640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-04-24T00:53:53.486 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:53.484+0000 7fbc2c221640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-04-24T00:53:53.697 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:53:53.696+0000 7fbc2ca22640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:53:53.698 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:53.696+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:53:53.817 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:53.816+0000 7efd52cd0640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:53:53.820 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:53:53.820+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:53:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:53:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:53:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:53:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap9-clean 2024-04-24T00:53:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-04-24T00:53:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:53:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9-clean 2024-04-24T00:53:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:53:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:53:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T00:53:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:53:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:53:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:53:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:53:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T00:53:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:53:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T00:53:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 54429696 bytes 2024-04-24T00:53:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 101398528 bytes 2024-04-24T00:54:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 121241600 bytes 2024-04-24T00:54:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 181825536 bytes 2024-04-24T00:54:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 222014464 bytes 2024-04-24T00:54:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:54:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T00:54:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:54:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T00:54:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T00:54:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:54:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:54:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T00:54:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T00:54:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:54:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T00:54:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":2218979.43,"entries_behind_primary":0,"entries_per_second":139.9,"non_primary_position":{"entry_tid":1,"object_number":121,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":121,"tag_tid":27}} 2024-04-24T00:54:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.3 on smithi134 2024-04-24T00:54:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:53:56 2024-04-24T00:54:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T00:54:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T00:54:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T00:54:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T00:54:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 00:53:57' 2024-04-24T00:54:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T00:54:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T00:54:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T00:54:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":2218979.43,"entries_behind_primary":0,"entries_per_second":139.9,"non_primary_position":{"entry_tid":1,"object_number":121,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":121,"tag_tid":27}} 2024-04-24T00:54:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:54:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:54:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T00:54:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:54:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T00:54:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.3 on smithi134 2024-04-24T00:54:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T00:54:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T00:54:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:53:56' 2024-04-24T00:54:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 00:53:56' 2024-04-24T00:54:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T00:54:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 00:53:56' 2024-04-24T00:54:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T00:54:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T00:54:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T00:54:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142194274mirror.3smithi13419.0.0-2424-gb98aab6ctrueOK42594312mirror.2smithi13419.0.0-2424-gb98aab6cfalseOK44534456mirror.0smithi13419.0.0-2424-gb98aab6cfalseOK45134516mirror.1smithi13419.0.0-2424-gb98aab6cfalseOKtestb94c2353-363b-4139-b9b9-403e9094bfc3up+replayingreplaying, {"bytes_per_second":2218979.43,"entries_behind_primary":0,"entries_per_second":139.9,"non_primary_position":{"entry_tid":1,"object_number":121,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":121,"tag_tid":27}}42194274mirror.3smithi1342024-04-24 00:53:56cluster2810b2a20-fcfd-4444-9ac3-7b8ecdd70280up+stoppedlocal image is primary2024-04-24 00:53:57' 2024-04-24T00:54:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T00:54:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T00:54:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 00:53:56' 2024-04-24T00:54:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T00:54:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T00:54:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T00:54:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T00:54:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T00:54:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10 2024-04-24T00:54:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap10 2024-04-24T00:54:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:54:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10 2024-04-24T00:54:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:54:11.807 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7403879483349602) > weight (0.25) 2024-04-24T00:54:11.807 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7672998691899233) > weight (0.25) 2024-04-24T00:54:11.807 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.47899161054954265) > weight (0.25) 2024-04-24T00:54:11.808 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9177622618725321) > weight (0.25) 2024-04-24T00:54:11.808 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-04-24T00:54:18.916 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6337696768673361) > weight (0.25) 2024-04-24T00:54:18.917 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9783784936124849) > weight (0.25) 2024-04-24T00:54:18.917 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5400104574983052) > weight (0.25) 2024-04-24T00:54:18.917 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.44164123330325955) > weight (0.25) 2024-04-24T00:54:18.917 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-24T00:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:54:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:54:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:54:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:54:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:54:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:54:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:54:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:54:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:54:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:54:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:54:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:54:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:54:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:54:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:54:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:54:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:54:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:54:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:54:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:54:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:54:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:54:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:54:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:54:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:54:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:54:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:54:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:54:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:54:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:54:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:54:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:54:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:54:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:54:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:54:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:54:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:54:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:54:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:54:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:54:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:54:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:54:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:54:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:54:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:54:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:54:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:54:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:54:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:54:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:54:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:54:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:54:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:54:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=128, tag_tid=29, entry_tid=6440], [object_number=131, tag_tid=29, entry_tid=6439], [object_number=130, tag_tid=29, entry_tid=6438], [object_number=129, tag_tid=29, entry_tid=6437]]], state=connected] 2024-04-24T00:54:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=128, tag_tid=29, entry_tid=6440], [object_number=131, tag_tid=29, entry_tid=6439], [object_number=130, tag_tid=29, entry_tid=6438], [object_number=129, tag_tid=29, entry_tid=6437]]], state=connected] 2024-04-24T00:54:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=128, tag_tid=29, entry_tid=6440' 2024-04-24T00:54:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=128, tag_tid=29, entry_tid=6440' 2024-04-24T00:54:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=128, tag_tid=29, entry_tid=6440' '!=' '' 2024-04-24T00:54:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=128, tag_tid=29, entry_tid=6440' = '' 2024-04-24T00:54:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=128, tag_tid=29, entry_tid=6440' 2024-04-24T00:54:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=128, tag_tid=29, entry_tid=6440' 2024-04-24T00:54:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=128, tag_tid=29, entry_tid=6440' 2024-04-24T00:54:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6440 2024-04-24T00:54:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 6440 2024-04-24T00:54:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=128, tag_tid=29, entry_tid=6608], [object_number=131, tag_tid=29, entry_tid=6607], [object_number=130, tag_tid=29, entry_tid=6606], [object_number=129, tag_tid=29, entry_tid=6605]]], state=connected] 2024-04-24T00:54:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=128, tag_tid=29, entry_tid=6608], [object_number=131, tag_tid=29, entry_tid=6607], [object_number=130, tag_tid=29, entry_tid=6606], [object_number=129, tag_tid=29, entry_tid=6605]]], state=connected] 2024-04-24T00:54:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=128, tag_tid=29, entry_tid=6608' 2024-04-24T00:54:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=128, tag_tid=29, entry_tid=6608' 2024-04-24T00:54:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=128, tag_tid=29, entry_tid=6608' '!=' 'object_number=128, tag_tid=29, entry_tid=6440' 2024-04-24T00:54:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=128, tag_tid=29, entry_tid=6608' = 'object_number=128, tag_tid=29, entry_tid=6440' 2024-04-24T00:54:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=128, tag_tid=29, entry_tid=6608' 2024-04-24T00:54:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=128, tag_tid=29, entry_tid=6608' 2024-04-24T00:54:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=128, tag_tid=29, entry_tid=6608' 2024-04-24T00:54:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6608 2024-04-24T00:54:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 6608 2024-04-24T00:54:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=131, tag_tid=29, entry_tid=6755], [object_number=130, tag_tid=29, entry_tid=6754], [object_number=129, tag_tid=29, entry_tid=6753], [object_number=128, tag_tid=29, entry_tid=6752]]], state=connected] 2024-04-24T00:54:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=131, tag_tid=29, entry_tid=6755], [object_number=130, tag_tid=29, entry_tid=6754], [object_number=129, tag_tid=29, entry_tid=6753], [object_number=128, tag_tid=29, entry_tid=6752]]], state=connected] 2024-04-24T00:54:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=131, tag_tid=29, entry_tid=6755' 2024-04-24T00:54:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=131, tag_tid=29, entry_tid=6755' 2024-04-24T00:54:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=131, tag_tid=29, entry_tid=6755' '!=' 'object_number=128, tag_tid=29, entry_tid=6608' 2024-04-24T00:54:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=131, tag_tid=29, entry_tid=6755' = 'object_number=128, tag_tid=29, entry_tid=6608' 2024-04-24T00:54:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=131, tag_tid=29, entry_tid=6755' 2024-04-24T00:54:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=131, tag_tid=29, entry_tid=6755' 2024-04-24T00:54:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=131, tag_tid=29, entry_tid=6755' 2024-04-24T00:54:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6755 2024-04-24T00:54:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 6755 2024-04-24T00:54:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=131, tag_tid=29, entry_tid=6943], [object_number=130, tag_tid=29, entry_tid=6942], [object_number=129, tag_tid=29, entry_tid=6941], [object_number=128, tag_tid=29, entry_tid=6940]]], state=connected] 2024-04-24T00:54:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:41.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=131, tag_tid=29, entry_tid=6943], [object_number=130, tag_tid=29, entry_tid=6942], [object_number=129, tag_tid=29, entry_tid=6941], [object_number=128, tag_tid=29, entry_tid=6940]]], state=connected] 2024-04-24T00:54:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=131, tag_tid=29, entry_tid=6943' 2024-04-24T00:54:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=131, tag_tid=29, entry_tid=6943' 2024-04-24T00:54:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=131, tag_tid=29, entry_tid=6943' '!=' 'object_number=131, tag_tid=29, entry_tid=6755' 2024-04-24T00:54:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=131, tag_tid=29, entry_tid=6943' = 'object_number=131, tag_tid=29, entry_tid=6755' 2024-04-24T00:54:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=131, tag_tid=29, entry_tid=6943' 2024-04-24T00:54:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=131, tag_tid=29, entry_tid=6943' 2024-04-24T00:54:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=131, tag_tid=29, entry_tid=6943' 2024-04-24T00:54:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6943 2024-04-24T00:54:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 6943 2024-04-24T00:54:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=128, tag_tid=29, entry_tid=7092], [object_number=131, tag_tid=29, entry_tid=7091], [object_number=130, tag_tid=29, entry_tid=7090], [object_number=129, tag_tid=29, entry_tid=7089]]], state=connected] 2024-04-24T00:54:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=128, tag_tid=29, entry_tid=7092], [object_number=131, tag_tid=29, entry_tid=7091], [object_number=130, tag_tid=29, entry_tid=7090], [object_number=129, tag_tid=29, entry_tid=7089]]], state=connected] 2024-04-24T00:54:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=128, tag_tid=29, entry_tid=7092' 2024-04-24T00:54:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=128, tag_tid=29, entry_tid=7092' 2024-04-24T00:54:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=128, tag_tid=29, entry_tid=7092' '!=' 'object_number=131, tag_tid=29, entry_tid=6943' 2024-04-24T00:54:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=128, tag_tid=29, entry_tid=7092' = 'object_number=131, tag_tid=29, entry_tid=6943' 2024-04-24T00:54:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=128, tag_tid=29, entry_tid=7092' 2024-04-24T00:54:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=128, tag_tid=29, entry_tid=7092' 2024-04-24T00:54:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=128, tag_tid=29, entry_tid=7092' 2024-04-24T00:54:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7092 2024-04-24T00:54:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 7092 2024-04-24T00:54:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:43.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:43.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:43.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=7241], [object_number=128, tag_tid=29, entry_tid=7240], [object_number=131, tag_tid=29, entry_tid=7239], [object_number=130, tag_tid=29, entry_tid=7238]]], state=connected] 2024-04-24T00:54:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=7241], [object_number=128, tag_tid=29, entry_tid=7240], [object_number=131, tag_tid=29, entry_tid=7239], [object_number=130, tag_tid=29, entry_tid=7238]]], state=connected] 2024-04-24T00:54:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=129, tag_tid=29, entry_tid=7241' 2024-04-24T00:54:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=129, tag_tid=29, entry_tid=7241' 2024-04-24T00:54:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=7241' '!=' 'object_number=128, tag_tid=29, entry_tid=7092' 2024-04-24T00:54:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=7241' = 'object_number=128, tag_tid=29, entry_tid=7092' 2024-04-24T00:54:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=129, tag_tid=29, entry_tid=7241' 2024-04-24T00:54:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=7241' 2024-04-24T00:54:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=7241' 2024-04-24T00:54:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7241 2024-04-24T00:54:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 7241 2024-04-24T00:54:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=7410], [object_number=129, tag_tid=29, entry_tid=7409], [object_number=128, tag_tid=29, entry_tid=7408], [object_number=131, tag_tid=29, entry_tid=7407]]], state=connected] 2024-04-24T00:54:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=7410], [object_number=129, tag_tid=29, entry_tid=7409], [object_number=128, tag_tid=29, entry_tid=7408], [object_number=131, tag_tid=29, entry_tid=7407]]], state=connected] 2024-04-24T00:54:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=130, tag_tid=29, entry_tid=7410' 2024-04-24T00:54:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=130, tag_tid=29, entry_tid=7410' 2024-04-24T00:54:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=7410' '!=' 'object_number=129, tag_tid=29, entry_tid=7241' 2024-04-24T00:54:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=7410' = 'object_number=129, tag_tid=29, entry_tid=7241' 2024-04-24T00:54:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=130, tag_tid=29, entry_tid=7410' 2024-04-24T00:54:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=7410' 2024-04-24T00:54:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=7410' 2024-04-24T00:54:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7410 2024-04-24T00:54:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 7410 2024-04-24T00:54:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=7581], [object_number=128, tag_tid=29, entry_tid=7580], [object_number=131, tag_tid=29, entry_tid=7579], [object_number=130, tag_tid=29, entry_tid=7578]]], state=connected] 2024-04-24T00:54:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=7581], [object_number=128, tag_tid=29, entry_tid=7580], [object_number=131, tag_tid=29, entry_tid=7579], [object_number=130, tag_tid=29, entry_tid=7578]]], state=connected] 2024-04-24T00:54:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=129, tag_tid=29, entry_tid=7581' 2024-04-24T00:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=129, tag_tid=29, entry_tid=7581' 2024-04-24T00:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=7581' '!=' 'object_number=130, tag_tid=29, entry_tid=7410' 2024-04-24T00:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=7581' = 'object_number=130, tag_tid=29, entry_tid=7410' 2024-04-24T00:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=129, tag_tid=29, entry_tid=7581' 2024-04-24T00:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=7581' 2024-04-24T00:54:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=7581' 2024-04-24T00:54:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7581 2024-04-24T00:54:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 7581 2024-04-24T00:54:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=7749], [object_number=128, tag_tid=29, entry_tid=7748], [object_number=131, tag_tid=29, entry_tid=7747], [object_number=130, tag_tid=29, entry_tid=7746]]], state=connected] 2024-04-24T00:54:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=7749], [object_number=128, tag_tid=29, entry_tid=7748], [object_number=131, tag_tid=29, entry_tid=7747], [object_number=130, tag_tid=29, entry_tid=7746]]], state=connected] 2024-04-24T00:54:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=129, tag_tid=29, entry_tid=7749' 2024-04-24T00:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=129, tag_tid=29, entry_tid=7749' 2024-04-24T00:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=7749' '!=' 'object_number=129, tag_tid=29, entry_tid=7581' 2024-04-24T00:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=7749' = 'object_number=129, tag_tid=29, entry_tid=7581' 2024-04-24T00:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=129, tag_tid=29, entry_tid=7749' 2024-04-24T00:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=7749' 2024-04-24T00:54:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=7749' 2024-04-24T00:54:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7749 2024-04-24T00:54:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 7749 2024-04-24T00:54:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=7918], [object_number=129, tag_tid=29, entry_tid=7917], [object_number=128, tag_tid=29, entry_tid=7916], [object_number=131, tag_tid=29, entry_tid=7915]]], state=connected] 2024-04-24T00:54:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=7918], [object_number=129, tag_tid=29, entry_tid=7917], [object_number=128, tag_tid=29, entry_tid=7916], [object_number=131, tag_tid=29, entry_tid=7915]]], state=connected] 2024-04-24T00:54:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=130, tag_tid=29, entry_tid=7918' 2024-04-24T00:54:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=130, tag_tid=29, entry_tid=7918' 2024-04-24T00:54:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=7918' '!=' 'object_number=129, tag_tid=29, entry_tid=7749' 2024-04-24T00:54:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=7918' = 'object_number=129, tag_tid=29, entry_tid=7749' 2024-04-24T00:54:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=130, tag_tid=29, entry_tid=7918' 2024-04-24T00:54:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=7918' 2024-04-24T00:54:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=7918' 2024-04-24T00:54:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7918 2024-04-24T00:54:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 7918 2024-04-24T00:54:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=8074], [object_number=129, tag_tid=29, entry_tid=8073], [object_number=128, tag_tid=29, entry_tid=8072], [object_number=131, tag_tid=29, entry_tid=8071]]], state=connected] 2024-04-24T00:54:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=8074], [object_number=129, tag_tid=29, entry_tid=8073], [object_number=128, tag_tid=29, entry_tid=8072], [object_number=131, tag_tid=29, entry_tid=8071]]], state=connected] 2024-04-24T00:54:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=130, tag_tid=29, entry_tid=8074' 2024-04-24T00:54:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=130, tag_tid=29, entry_tid=8074' 2024-04-24T00:54:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=8074' '!=' 'object_number=130, tag_tid=29, entry_tid=7918' 2024-04-24T00:54:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=8074' = 'object_number=130, tag_tid=29, entry_tid=7918' 2024-04-24T00:54:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=130, tag_tid=29, entry_tid=8074' 2024-04-24T00:54:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=8074' 2024-04-24T00:54:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=8074' 2024-04-24T00:54:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8074 2024-04-24T00:54:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 8074 2024-04-24T00:54:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=8230], [object_number=129, tag_tid=29, entry_tid=8229], [object_number=128, tag_tid=29, entry_tid=8228], [object_number=131, tag_tid=29, entry_tid=8227]]], state=connected] 2024-04-24T00:54:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=8230], [object_number=129, tag_tid=29, entry_tid=8229], [object_number=128, tag_tid=29, entry_tid=8228], [object_number=131, tag_tid=29, entry_tid=8227]]], state=connected] 2024-04-24T00:54:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=130, tag_tid=29, entry_tid=8230' 2024-04-24T00:54:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=130, tag_tid=29, entry_tid=8230' 2024-04-24T00:54:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=8230' '!=' 'object_number=130, tag_tid=29, entry_tid=8074' 2024-04-24T00:54:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=8230' = 'object_number=130, tag_tid=29, entry_tid=8074' 2024-04-24T00:54:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=130, tag_tid=29, entry_tid=8230' 2024-04-24T00:54:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=8230' 2024-04-24T00:54:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=8230' 2024-04-24T00:54:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8230 2024-04-24T00:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 8230 2024-04-24T00:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=8381], [object_number=128, tag_tid=29, entry_tid=8380], [object_number=131, tag_tid=29, entry_tid=8379], [object_number=130, tag_tid=29, entry_tid=8378]]], state=connected] 2024-04-24T00:54:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=8381], [object_number=128, tag_tid=29, entry_tid=8380], [object_number=131, tag_tid=29, entry_tid=8379], [object_number=130, tag_tid=29, entry_tid=8378]]], state=connected] 2024-04-24T00:54:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=129, tag_tid=29, entry_tid=8381' 2024-04-24T00:54:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=129, tag_tid=29, entry_tid=8381' 2024-04-24T00:54:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=8381' '!=' 'object_number=130, tag_tid=29, entry_tid=8230' 2024-04-24T00:54:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=8381' = 'object_number=130, tag_tid=29, entry_tid=8230' 2024-04-24T00:54:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=129, tag_tid=29, entry_tid=8381' 2024-04-24T00:54:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=8381' 2024-04-24T00:54:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=8381' 2024-04-24T00:54:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8381 2024-04-24T00:54:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 8381 2024-04-24T00:54:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=8526], [object_number=129, tag_tid=29, entry_tid=8525], [object_number=128, tag_tid=29, entry_tid=8524], [object_number=131, tag_tid=29, entry_tid=8523]]], state=connected] 2024-04-24T00:54:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=130, tag_tid=29, entry_tid=8526], [object_number=129, tag_tid=29, entry_tid=8525], [object_number=128, tag_tid=29, entry_tid=8524], [object_number=131, tag_tid=29, entry_tid=8523]]], state=connected] 2024-04-24T00:54:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=130, tag_tid=29, entry_tid=8526' 2024-04-24T00:54:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=130, tag_tid=29, entry_tid=8526' 2024-04-24T00:54:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=8526' '!=' 'object_number=129, tag_tid=29, entry_tid=8381' 2024-04-24T00:54:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=130, tag_tid=29, entry_tid=8526' = 'object_number=129, tag_tid=29, entry_tid=8381' 2024-04-24T00:54:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=130, tag_tid=29, entry_tid=8526' 2024-04-24T00:54:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=8526' 2024-04-24T00:54:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=130, tag_tid=29, entry_tid=8526' 2024-04-24T00:54:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8526 2024-04-24T00:54:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 8526 2024-04-24T00:54:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=8685], [object_number=128, tag_tid=29, entry_tid=8684], [object_number=131, tag_tid=29, entry_tid=8683], [object_number=130, tag_tid=29, entry_tid=8682]]], state=connected] 2024-04-24T00:54:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=8685], [object_number=128, tag_tid=29, entry_tid=8684], [object_number=131, tag_tid=29, entry_tid=8683], [object_number=130, tag_tid=29, entry_tid=8682]]], state=connected] 2024-04-24T00:54:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=129, tag_tid=29, entry_tid=8685' 2024-04-24T00:54:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=129, tag_tid=29, entry_tid=8685' 2024-04-24T00:54:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=8685' '!=' 'object_number=130, tag_tid=29, entry_tid=8526' 2024-04-24T00:54:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=8685' = 'object_number=130, tag_tid=29, entry_tid=8526' 2024-04-24T00:54:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=129, tag_tid=29, entry_tid=8685' 2024-04-24T00:54:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=8685' 2024-04-24T00:54:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=8685' 2024-04-24T00:54:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8685 2024-04-24T00:54:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 8685 2024-04-24T00:54:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:52.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:52.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:52.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:52.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=128, tag_tid=29, entry_tid=8836], [object_number=131, tag_tid=29, entry_tid=8835], [object_number=130, tag_tid=29, entry_tid=8834], [object_number=129, tag_tid=29, entry_tid=8833]]], state=connected] 2024-04-24T00:54:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=128, tag_tid=29, entry_tid=8836], [object_number=131, tag_tid=29, entry_tid=8835], [object_number=130, tag_tid=29, entry_tid=8834], [object_number=129, tag_tid=29, entry_tid=8833]]], state=connected] 2024-04-24T00:54:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=128, tag_tid=29, entry_tid=8836' 2024-04-24T00:54:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=128, tag_tid=29, entry_tid=8836' 2024-04-24T00:54:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=128, tag_tid=29, entry_tid=8836' '!=' 'object_number=129, tag_tid=29, entry_tid=8685' 2024-04-24T00:54:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=128, tag_tid=29, entry_tid=8836' = 'object_number=129, tag_tid=29, entry_tid=8685' 2024-04-24T00:54:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=128, tag_tid=29, entry_tid=8836' 2024-04-24T00:54:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=128, tag_tid=29, entry_tid=8836' 2024-04-24T00:54:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=128, tag_tid=29, entry_tid=8836' 2024-04-24T00:54:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8836 2024-04-24T00:54:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 8836 2024-04-24T00:54:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=8993], [object_number=128, tag_tid=29, entry_tid=8992], [object_number=131, tag_tid=29, entry_tid=8991], [object_number=130, tag_tid=29, entry_tid=8990]]], state=connected] 2024-04-24T00:54:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=8993], [object_number=128, tag_tid=29, entry_tid=8992], [object_number=131, tag_tid=29, entry_tid=8991], [object_number=130, tag_tid=29, entry_tid=8990]]], state=connected] 2024-04-24T00:54:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=129, tag_tid=29, entry_tid=8993' 2024-04-24T00:54:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=129, tag_tid=29, entry_tid=8993' 2024-04-24T00:54:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=8993' '!=' 'object_number=128, tag_tid=29, entry_tid=8836' 2024-04-24T00:54:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=8993' = 'object_number=128, tag_tid=29, entry_tid=8836' 2024-04-24T00:54:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=129, tag_tid=29, entry_tid=8993' 2024-04-24T00:54:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=8993' 2024-04-24T00:54:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=8993' 2024-04-24T00:54:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8993 2024-04-24T00:54:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 8993 2024-04-24T00:54:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:53.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:53.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:53.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=9153], [object_number=128, tag_tid=29, entry_tid=9152], [object_number=131, tag_tid=29, entry_tid=9151], [object_number=130, tag_tid=29, entry_tid=9150]]], state=connected] 2024-04-24T00:54:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 32 2024-04-24T00:54:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=129, tag_tid=29, entry_tid=9153], [object_number=128, tag_tid=29, entry_tid=9152], [object_number=131, tag_tid=29, entry_tid=9151], [object_number=130, tag_tid=29, entry_tid=9150]]], state=connected] 2024-04-24T00:54:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=129, tag_tid=29, entry_tid=9153' 2024-04-24T00:54:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=129, tag_tid=29, entry_tid=9153' 2024-04-24T00:54:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=9153' '!=' 'object_number=129, tag_tid=29, entry_tid=8993' 2024-04-24T00:54:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=129, tag_tid=29, entry_tid=9153' = 'object_number=129, tag_tid=29, entry_tid=8993' 2024-04-24T00:54:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=129, tag_tid=29, entry_tid=9153' 2024-04-24T00:54:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=9153' 2024-04-24T00:54:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=129, tag_tid=29, entry_tid=9153' 2024-04-24T00:54:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9153 2024-04-24T00:54:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 9153 2024-04-24T00:54:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=9296], [object_number=135, tag_tid=29, entry_tid=9295], [object_number=134, tag_tid=29, entry_tid=9294], [object_number=133, tag_tid=29, entry_tid=9293]]], state=connected] 2024-04-24T00:54:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=9296], [object_number=135, tag_tid=29, entry_tid=9295], [object_number=134, tag_tid=29, entry_tid=9294], [object_number=133, tag_tid=29, entry_tid=9293]]], state=connected] 2024-04-24T00:54:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=132, tag_tid=29, entry_tid=9296' 2024-04-24T00:54:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=132, tag_tid=29, entry_tid=9296' 2024-04-24T00:54:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=9296' '!=' 'object_number=129, tag_tid=29, entry_tid=9153' 2024-04-24T00:54:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=9296' = 'object_number=129, tag_tid=29, entry_tid=9153' 2024-04-24T00:54:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=132, tag_tid=29, entry_tid=9296' 2024-04-24T00:54:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=9296' 2024-04-24T00:54:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=9296' 2024-04-24T00:54:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9296 2024-04-24T00:54:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 9296 2024-04-24T00:54:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=9482], [object_number=133, tag_tid=29, entry_tid=9481], [object_number=132, tag_tid=29, entry_tid=9480], [object_number=135, tag_tid=29, entry_tid=9479]]], state=connected] 2024-04-24T00:54:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=9482], [object_number=133, tag_tid=29, entry_tid=9481], [object_number=132, tag_tid=29, entry_tid=9480], [object_number=135, tag_tid=29, entry_tid=9479]]], state=connected] 2024-04-24T00:54:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=134, tag_tid=29, entry_tid=9482' 2024-04-24T00:54:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=134, tag_tid=29, entry_tid=9482' 2024-04-24T00:54:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=9482' '!=' 'object_number=132, tag_tid=29, entry_tid=9296' 2024-04-24T00:54:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=9482' = 'object_number=132, tag_tid=29, entry_tid=9296' 2024-04-24T00:54:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=134, tag_tid=29, entry_tid=9482' 2024-04-24T00:54:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=9482' 2024-04-24T00:54:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=9482' 2024-04-24T00:54:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9482 2024-04-24T00:54:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 9482 2024-04-24T00:54:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=9678], [object_number=133, tag_tid=29, entry_tid=9677], [object_number=132, tag_tid=29, entry_tid=9676], [object_number=135, tag_tid=29, entry_tid=9675]]], state=connected] 2024-04-24T00:54:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=9678], [object_number=133, tag_tid=29, entry_tid=9677], [object_number=132, tag_tid=29, entry_tid=9676], [object_number=135, tag_tid=29, entry_tid=9675]]], state=connected] 2024-04-24T00:54:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=134, tag_tid=29, entry_tid=9678' 2024-04-24T00:54:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=134, tag_tid=29, entry_tid=9678' 2024-04-24T00:54:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=9678' '!=' 'object_number=134, tag_tid=29, entry_tid=9482' 2024-04-24T00:54:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=9678' = 'object_number=134, tag_tid=29, entry_tid=9482' 2024-04-24T00:54:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=134, tag_tid=29, entry_tid=9678' 2024-04-24T00:54:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=9678' 2024-04-24T00:54:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=9678' 2024-04-24T00:54:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9678 2024-04-24T00:54:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 9678 2024-04-24T00:54:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=9838], [object_number=133, tag_tid=29, entry_tid=9837], [object_number=132, tag_tid=29, entry_tid=9836], [object_number=135, tag_tid=29, entry_tid=9835]]], state=connected] 2024-04-24T00:54:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=9838], [object_number=133, tag_tid=29, entry_tid=9837], [object_number=132, tag_tid=29, entry_tid=9836], [object_number=135, tag_tid=29, entry_tid=9835]]], state=connected] 2024-04-24T00:54:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=134, tag_tid=29, entry_tid=9838' 2024-04-24T00:54:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=134, tag_tid=29, entry_tid=9838' 2024-04-24T00:54:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=9838' '!=' 'object_number=134, tag_tid=29, entry_tid=9678' 2024-04-24T00:54:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=9838' = 'object_number=134, tag_tid=29, entry_tid=9678' 2024-04-24T00:54:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=134, tag_tid=29, entry_tid=9838' 2024-04-24T00:54:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=9838' 2024-04-24T00:54:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=9838' 2024-04-24T00:54:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9838 2024-04-24T00:54:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 9838 2024-04-24T00:54:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=9989], [object_number=132, tag_tid=29, entry_tid=9988], [object_number=135, tag_tid=29, entry_tid=9987], [object_number=134, tag_tid=29, entry_tid=9986]]], state=connected] 2024-04-24T00:54:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=9989], [object_number=132, tag_tid=29, entry_tid=9988], [object_number=135, tag_tid=29, entry_tid=9987], [object_number=134, tag_tid=29, entry_tid=9986]]], state=connected] 2024-04-24T00:54:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=133, tag_tid=29, entry_tid=9989' 2024-04-24T00:54:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=133, tag_tid=29, entry_tid=9989' 2024-04-24T00:54:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=9989' '!=' 'object_number=134, tag_tid=29, entry_tid=9838' 2024-04-24T00:54:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=9989' = 'object_number=134, tag_tid=29, entry_tid=9838' 2024-04-24T00:54:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=133, tag_tid=29, entry_tid=9989' 2024-04-24T00:54:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=9989' 2024-04-24T00:54:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=9989' 2024-04-24T00:54:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9989 2024-04-24T00:54:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 9989 2024-04-24T00:54:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:54:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:54:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:54:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:54:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:54:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:54:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:54:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:54:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:54:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:54:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:54:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:54:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:54:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:54:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:54:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=10136], [object_number=135, tag_tid=29, entry_tid=10135], [object_number=134, tag_tid=29, entry_tid=10134], [object_number=133, tag_tid=29, entry_tid=10133]]], state=connected] 2024-04-24T00:54:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:54:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:54:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:54:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:54:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:54:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:54:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:54:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:54:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=10136], [object_number=135, tag_tid=29, entry_tid=10135], [object_number=134, tag_tid=29, entry_tid=10134], [object_number=133, tag_tid=29, entry_tid=10133]]], state=connected] 2024-04-24T00:54:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:54:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=132, tag_tid=29, entry_tid=10136' 2024-04-24T00:54:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=132, tag_tid=29, entry_tid=10136' 2024-04-24T00:54:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=10136' '!=' 'object_number=133, tag_tid=29, entry_tid=9989' 2024-04-24T00:54:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:54:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=10136' = 'object_number=133, tag_tid=29, entry_tid=9989' 2024-04-24T00:54:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=132, tag_tid=29, entry_tid=10136' 2024-04-24T00:54:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:54:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=10136' 2024-04-24T00:54:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:54:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:54:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:54:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:54:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=10136' 2024-04-24T00:54:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:54:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:54:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10136 2024-04-24T00:54:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 10136 2024-04-24T00:54:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:54:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:54:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:54:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:54:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:54:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:54:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:54:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:54:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:54:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:54:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:54:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:54:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:54:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:54:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:54:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:54:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:54:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:54:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:54:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:54:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:54:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:54:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:54:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:54:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:54:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:54:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:54:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:54:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:54:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:54:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:54:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:54:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:54:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:54:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:54:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:54:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:54:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:54:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:54:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:54:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:54:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:54:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:54:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:54:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:54:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:54:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=10285], [object_number=132, tag_tid=29, entry_tid=10284], [object_number=135, tag_tid=29, entry_tid=10283], [object_number=134, tag_tid=29, entry_tid=10282]]], state=connected] 2024-04-24T00:55:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=10285], [object_number=132, tag_tid=29, entry_tid=10284], [object_number=135, tag_tid=29, entry_tid=10283], [object_number=134, tag_tid=29, entry_tid=10282]]], state=connected] 2024-04-24T00:55:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=133, tag_tid=29, entry_tid=10285' 2024-04-24T00:55:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=133, tag_tid=29, entry_tid=10285' 2024-04-24T00:55:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=10285' '!=' 'object_number=132, tag_tid=29, entry_tid=10136' 2024-04-24T00:55:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=10285' = 'object_number=132, tag_tid=29, entry_tid=10136' 2024-04-24T00:55:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=133, tag_tid=29, entry_tid=10285' 2024-04-24T00:55:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=10285' 2024-04-24T00:55:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=10285' 2024-04-24T00:55:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10285 2024-04-24T00:55:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 10285 2024-04-24T00:55:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:00.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=10433], [object_number=132, tag_tid=29, entry_tid=10432], [object_number=135, tag_tid=29, entry_tid=10431], [object_number=134, tag_tid=29, entry_tid=10430]]], state=connected] 2024-04-24T00:55:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=10433], [object_number=132, tag_tid=29, entry_tid=10432], [object_number=135, tag_tid=29, entry_tid=10431], [object_number=134, tag_tid=29, entry_tid=10430]]], state=connected] 2024-04-24T00:55:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=133, tag_tid=29, entry_tid=10433' 2024-04-24T00:55:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=133, tag_tid=29, entry_tid=10433' 2024-04-24T00:55:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=10433' '!=' 'object_number=133, tag_tid=29, entry_tid=10285' 2024-04-24T00:55:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=10433' = 'object_number=133, tag_tid=29, entry_tid=10285' 2024-04-24T00:55:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=133, tag_tid=29, entry_tid=10433' 2024-04-24T00:55:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=10433' 2024-04-24T00:55:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=10433' 2024-04-24T00:55:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10433 2024-04-24T00:55:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 10433 2024-04-24T00:55:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=10582], [object_number=133, tag_tid=29, entry_tid=10581], [object_number=132, tag_tid=29, entry_tid=10580], [object_number=135, tag_tid=29, entry_tid=10579]]], state=connected] 2024-04-24T00:55:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=10582], [object_number=133, tag_tid=29, entry_tid=10581], [object_number=132, tag_tid=29, entry_tid=10580], [object_number=135, tag_tid=29, entry_tid=10579]]], state=connected] 2024-04-24T00:55:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=134, tag_tid=29, entry_tid=10582' 2024-04-24T00:55:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=134, tag_tid=29, entry_tid=10582' 2024-04-24T00:55:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=10582' '!=' 'object_number=133, tag_tid=29, entry_tid=10433' 2024-04-24T00:55:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=10582' = 'object_number=133, tag_tid=29, entry_tid=10433' 2024-04-24T00:55:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=134, tag_tid=29, entry_tid=10582' 2024-04-24T00:55:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=10582' 2024-04-24T00:55:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=10582' 2024-04-24T00:55:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10582 2024-04-24T00:55:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 10582 2024-04-24T00:55:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=10738], [object_number=133, tag_tid=29, entry_tid=10737], [object_number=132, tag_tid=29, entry_tid=10736], [object_number=135, tag_tid=29, entry_tid=10735]]], state=connected] 2024-04-24T00:55:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=10738], [object_number=133, tag_tid=29, entry_tid=10737], [object_number=132, tag_tid=29, entry_tid=10736], [object_number=135, tag_tid=29, entry_tid=10735]]], state=connected] 2024-04-24T00:55:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=134, tag_tid=29, entry_tid=10738' 2024-04-24T00:55:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=134, tag_tid=29, entry_tid=10738' 2024-04-24T00:55:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=10738' '!=' 'object_number=134, tag_tid=29, entry_tid=10582' 2024-04-24T00:55:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=10738' = 'object_number=134, tag_tid=29, entry_tid=10582' 2024-04-24T00:55:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=134, tag_tid=29, entry_tid=10738' 2024-04-24T00:55:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=10738' 2024-04-24T00:55:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=10738' 2024-04-24T00:55:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10738 2024-04-24T00:55:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 10738 2024-04-24T00:55:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=10909], [object_number=132, tag_tid=29, entry_tid=10908], [object_number=135, tag_tid=29, entry_tid=10907], [object_number=134, tag_tid=29, entry_tid=10906]]], state=connected] 2024-04-24T00:55:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=10909], [object_number=132, tag_tid=29, entry_tid=10908], [object_number=135, tag_tid=29, entry_tid=10907], [object_number=134, tag_tid=29, entry_tid=10906]]], state=connected] 2024-04-24T00:55:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=133, tag_tid=29, entry_tid=10909' 2024-04-24T00:55:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=133, tag_tid=29, entry_tid=10909' 2024-04-24T00:55:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=10909' '!=' 'object_number=134, tag_tid=29, entry_tid=10738' 2024-04-24T00:55:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=10909' = 'object_number=134, tag_tid=29, entry_tid=10738' 2024-04-24T00:55:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=133, tag_tid=29, entry_tid=10909' 2024-04-24T00:55:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=10909' 2024-04-24T00:55:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=10909' 2024-04-24T00:55:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10909 2024-04-24T00:55:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 10909 2024-04-24T00:55:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=11079], [object_number=134, tag_tid=29, entry_tid=11078], [object_number=133, tag_tid=29, entry_tid=11077], [object_number=132, tag_tid=29, entry_tid=11076]]], state=connected] 2024-04-24T00:55:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=11079], [object_number=134, tag_tid=29, entry_tid=11078], [object_number=133, tag_tid=29, entry_tid=11077], [object_number=132, tag_tid=29, entry_tid=11076]]], state=connected] 2024-04-24T00:55:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=135, tag_tid=29, entry_tid=11079' 2024-04-24T00:55:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=135, tag_tid=29, entry_tid=11079' 2024-04-24T00:55:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=11079' '!=' 'object_number=133, tag_tid=29, entry_tid=10909' 2024-04-24T00:55:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=11079' = 'object_number=133, tag_tid=29, entry_tid=10909' 2024-04-24T00:55:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=135, tag_tid=29, entry_tid=11079' 2024-04-24T00:55:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=11079' 2024-04-24T00:55:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=11079' 2024-04-24T00:55:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11079 2024-04-24T00:55:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 11079 2024-04-24T00:55:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=11244], [object_number=135, tag_tid=29, entry_tid=11243], [object_number=134, tag_tid=29, entry_tid=11242], [object_number=133, tag_tid=29, entry_tid=11241]]], state=connected] 2024-04-24T00:55:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=11244], [object_number=135, tag_tid=29, entry_tid=11243], [object_number=134, tag_tid=29, entry_tid=11242], [object_number=133, tag_tid=29, entry_tid=11241]]], state=connected] 2024-04-24T00:55:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=132, tag_tid=29, entry_tid=11244' 2024-04-24T00:55:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=132, tag_tid=29, entry_tid=11244' 2024-04-24T00:55:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=11244' '!=' 'object_number=135, tag_tid=29, entry_tid=11079' 2024-04-24T00:55:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=11244' = 'object_number=135, tag_tid=29, entry_tid=11079' 2024-04-24T00:55:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=132, tag_tid=29, entry_tid=11244' 2024-04-24T00:55:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=11244' 2024-04-24T00:55:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=11244' 2024-04-24T00:55:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11244 2024-04-24T00:55:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 11244 2024-04-24T00:55:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:06.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=11399], [object_number=134, tag_tid=29, entry_tid=11398], [object_number=133, tag_tid=29, entry_tid=11397], [object_number=132, tag_tid=29, entry_tid=11396]]], state=connected] 2024-04-24T00:55:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=11399], [object_number=134, tag_tid=29, entry_tid=11398], [object_number=133, tag_tid=29, entry_tid=11397], [object_number=132, tag_tid=29, entry_tid=11396]]], state=connected] 2024-04-24T00:55:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=135, tag_tid=29, entry_tid=11399' 2024-04-24T00:55:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=135, tag_tid=29, entry_tid=11399' 2024-04-24T00:55:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=11399' '!=' 'object_number=132, tag_tid=29, entry_tid=11244' 2024-04-24T00:55:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=11399' = 'object_number=132, tag_tid=29, entry_tid=11244' 2024-04-24T00:55:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=135, tag_tid=29, entry_tid=11399' 2024-04-24T00:55:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=11399' 2024-04-24T00:55:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=11399' 2024-04-24T00:55:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11399 2024-04-24T00:55:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 11399 2024-04-24T00:55:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=11562], [object_number=133, tag_tid=29, entry_tid=11561], [object_number=132, tag_tid=29, entry_tid=11560], [object_number=135, tag_tid=29, entry_tid=11559]]], state=connected] 2024-04-24T00:55:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=11562], [object_number=133, tag_tid=29, entry_tid=11561], [object_number=132, tag_tid=29, entry_tid=11560], [object_number=135, tag_tid=29, entry_tid=11559]]], state=connected] 2024-04-24T00:55:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=134, tag_tid=29, entry_tid=11562' 2024-04-24T00:55:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=134, tag_tid=29, entry_tid=11562' 2024-04-24T00:55:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=11562' '!=' 'object_number=135, tag_tid=29, entry_tid=11399' 2024-04-24T00:55:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=11562' = 'object_number=135, tag_tid=29, entry_tid=11399' 2024-04-24T00:55:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=134, tag_tid=29, entry_tid=11562' 2024-04-24T00:55:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=11562' 2024-04-24T00:55:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=11562' 2024-04-24T00:55:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11562 2024-04-24T00:55:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 11562 2024-04-24T00:55:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=11728], [object_number=135, tag_tid=29, entry_tid=11727], [object_number=134, tag_tid=29, entry_tid=11726], [object_number=133, tag_tid=29, entry_tid=11725]]], state=connected] 2024-04-24T00:55:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=11728], [object_number=135, tag_tid=29, entry_tid=11727], [object_number=134, tag_tid=29, entry_tid=11726], [object_number=133, tag_tid=29, entry_tid=11725]]], state=connected] 2024-04-24T00:55:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=132, tag_tid=29, entry_tid=11728' 2024-04-24T00:55:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=132, tag_tid=29, entry_tid=11728' 2024-04-24T00:55:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=11728' '!=' 'object_number=134, tag_tid=29, entry_tid=11562' 2024-04-24T00:55:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=11728' = 'object_number=134, tag_tid=29, entry_tid=11562' 2024-04-24T00:55:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=132, tag_tid=29, entry_tid=11728' 2024-04-24T00:55:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=11728' 2024-04-24T00:55:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=11728' 2024-04-24T00:55:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11728 2024-04-24T00:55:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 11728 2024-04-24T00:55:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=11874], [object_number=133, tag_tid=29, entry_tid=11873], [object_number=132, tag_tid=29, entry_tid=11872], [object_number=135, tag_tid=29, entry_tid=11871]]], state=connected] 2024-04-24T00:55:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=11874], [object_number=133, tag_tid=29, entry_tid=11873], [object_number=132, tag_tid=29, entry_tid=11872], [object_number=135, tag_tid=29, entry_tid=11871]]], state=connected] 2024-04-24T00:55:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=134, tag_tid=29, entry_tid=11874' 2024-04-24T00:55:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=134, tag_tid=29, entry_tid=11874' 2024-04-24T00:55:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=11874' '!=' 'object_number=132, tag_tid=29, entry_tid=11728' 2024-04-24T00:55:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=11874' = 'object_number=132, tag_tid=29, entry_tid=11728' 2024-04-24T00:55:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=134, tag_tid=29, entry_tid=11874' 2024-04-24T00:55:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=11874' 2024-04-24T00:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=11874' 2024-04-24T00:55:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11874 2024-04-24T00:55:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 11874 2024-04-24T00:55:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=12020], [object_number=135, tag_tid=29, entry_tid=12019], [object_number=134, tag_tid=29, entry_tid=12018], [object_number=133, tag_tid=29, entry_tid=12017]]], state=connected] 2024-04-24T00:55:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=12020], [object_number=135, tag_tid=29, entry_tid=12019], [object_number=134, tag_tid=29, entry_tid=12018], [object_number=133, tag_tid=29, entry_tid=12017]]], state=connected] 2024-04-24T00:55:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=132, tag_tid=29, entry_tid=12020' 2024-04-24T00:55:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=132, tag_tid=29, entry_tid=12020' 2024-04-24T00:55:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=12020' '!=' 'object_number=134, tag_tid=29, entry_tid=11874' 2024-04-24T00:55:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=12020' = 'object_number=134, tag_tid=29, entry_tid=11874' 2024-04-24T00:55:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=132, tag_tid=29, entry_tid=12020' 2024-04-24T00:55:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=12020' 2024-04-24T00:55:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=12020' 2024-04-24T00:55:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12020 2024-04-24T00:55:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 12020 2024-04-24T00:55:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=12172], [object_number=135, tag_tid=29, entry_tid=12171], [object_number=134, tag_tid=29, entry_tid=12170], [object_number=133, tag_tid=29, entry_tid=12169]]], state=connected] 2024-04-24T00:55:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=12172], [object_number=135, tag_tid=29, entry_tid=12171], [object_number=134, tag_tid=29, entry_tid=12170], [object_number=133, tag_tid=29, entry_tid=12169]]], state=connected] 2024-04-24T00:55:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=132, tag_tid=29, entry_tid=12172' 2024-04-24T00:55:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=132, tag_tid=29, entry_tid=12172' 2024-04-24T00:55:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=12172' '!=' 'object_number=132, tag_tid=29, entry_tid=12020' 2024-04-24T00:55:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=12172' = 'object_number=132, tag_tid=29, entry_tid=12020' 2024-04-24T00:55:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=132, tag_tid=29, entry_tid=12172' 2024-04-24T00:55:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=12172' 2024-04-24T00:55:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=12172' 2024-04-24T00:55:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12172 2024-04-24T00:55:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 12172 2024-04-24T00:55:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=12314], [object_number=133, tag_tid=29, entry_tid=12313], [object_number=132, tag_tid=29, entry_tid=12312], [object_number=135, tag_tid=29, entry_tid=12311]]], state=connected] 2024-04-24T00:55:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=134, tag_tid=29, entry_tid=12314], [object_number=133, tag_tid=29, entry_tid=12313], [object_number=132, tag_tid=29, entry_tid=12312], [object_number=135, tag_tid=29, entry_tid=12311]]], state=connected] 2024-04-24T00:55:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=134, tag_tid=29, entry_tid=12314' 2024-04-24T00:55:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=134, tag_tid=29, entry_tid=12314' 2024-04-24T00:55:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=12314' '!=' 'object_number=132, tag_tid=29, entry_tid=12172' 2024-04-24T00:55:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=134, tag_tid=29, entry_tid=12314' = 'object_number=132, tag_tid=29, entry_tid=12172' 2024-04-24T00:55:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=134, tag_tid=29, entry_tid=12314' 2024-04-24T00:55:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=12314' 2024-04-24T00:55:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=134, tag_tid=29, entry_tid=12314' 2024-04-24T00:55:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12314 2024-04-24T00:55:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 12314 2024-04-24T00:55:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=12461], [object_number=132, tag_tid=29, entry_tid=12460], [object_number=135, tag_tid=29, entry_tid=12459], [object_number=134, tag_tid=29, entry_tid=12458]]], state=connected] 2024-04-24T00:55:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=12461], [object_number=132, tag_tid=29, entry_tid=12460], [object_number=135, tag_tid=29, entry_tid=12459], [object_number=134, tag_tid=29, entry_tid=12458]]], state=connected] 2024-04-24T00:55:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=133, tag_tid=29, entry_tid=12461' 2024-04-24T00:55:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=133, tag_tid=29, entry_tid=12461' 2024-04-24T00:55:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=12461' '!=' 'object_number=134, tag_tid=29, entry_tid=12314' 2024-04-24T00:55:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=12461' = 'object_number=134, tag_tid=29, entry_tid=12314' 2024-04-24T00:55:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=133, tag_tid=29, entry_tid=12461' 2024-04-24T00:55:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=12461' 2024-04-24T00:55:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=12461' 2024-04-24T00:55:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12461 2024-04-24T00:55:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 12461 2024-04-24T00:55:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=12615], [object_number=134, tag_tid=29, entry_tid=12614], [object_number=133, tag_tid=29, entry_tid=12613], [object_number=132, tag_tid=29, entry_tid=12612]]], state=connected] 2024-04-24T00:55:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=12615], [object_number=134, tag_tid=29, entry_tid=12614], [object_number=133, tag_tid=29, entry_tid=12613], [object_number=132, tag_tid=29, entry_tid=12612]]], state=connected] 2024-04-24T00:55:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=135, tag_tid=29, entry_tid=12615' 2024-04-24T00:55:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=135, tag_tid=29, entry_tid=12615' 2024-04-24T00:55:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=12615' '!=' 'object_number=133, tag_tid=29, entry_tid=12461' 2024-04-24T00:55:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=12615' = 'object_number=133, tag_tid=29, entry_tid=12461' 2024-04-24T00:55:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=135, tag_tid=29, entry_tid=12615' 2024-04-24T00:55:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=12615' 2024-04-24T00:55:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=12615' 2024-04-24T00:55:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12615 2024-04-24T00:55:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 12615 2024-04-24T00:55:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=12771], [object_number=134, tag_tid=29, entry_tid=12770], [object_number=133, tag_tid=29, entry_tid=12769], [object_number=132, tag_tid=29, entry_tid=12768]]], state=connected] 2024-04-24T00:55:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=12771], [object_number=134, tag_tid=29, entry_tid=12770], [object_number=133, tag_tid=29, entry_tid=12769], [object_number=132, tag_tid=29, entry_tid=12768]]], state=connected] 2024-04-24T00:55:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=135, tag_tid=29, entry_tid=12771' 2024-04-24T00:55:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=135, tag_tid=29, entry_tid=12771' 2024-04-24T00:55:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=12771' '!=' 'object_number=135, tag_tid=29, entry_tid=12615' 2024-04-24T00:55:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=12771' = 'object_number=135, tag_tid=29, entry_tid=12615' 2024-04-24T00:55:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=135, tag_tid=29, entry_tid=12771' 2024-04-24T00:55:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=12771' 2024-04-24T00:55:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=12771' 2024-04-24T00:55:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12771 2024-04-24T00:55:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 12771 2024-04-24T00:55:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=12935], [object_number=134, tag_tid=29, entry_tid=12934], [object_number=133, tag_tid=29, entry_tid=12933], [object_number=132, tag_tid=29, entry_tid=12932]]], state=connected] 2024-04-24T00:55:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=12935], [object_number=134, tag_tid=29, entry_tid=12934], [object_number=133, tag_tid=29, entry_tid=12933], [object_number=132, tag_tid=29, entry_tid=12932]]], state=connected] 2024-04-24T00:55:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=135, tag_tid=29, entry_tid=12935' 2024-04-24T00:55:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=135, tag_tid=29, entry_tid=12935' 2024-04-24T00:55:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=12935' '!=' 'object_number=135, tag_tid=29, entry_tid=12771' 2024-04-24T00:55:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=12935' = 'object_number=135, tag_tid=29, entry_tid=12771' 2024-04-24T00:55:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=135, tag_tid=29, entry_tid=12935' 2024-04-24T00:55:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=12935' 2024-04-24T00:55:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=12935' 2024-04-24T00:55:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12935 2024-04-24T00:55:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 12935 2024-04-24T00:55:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=13104], [object_number=135, tag_tid=29, entry_tid=13103], [object_number=134, tag_tid=29, entry_tid=13102], [object_number=133, tag_tid=29, entry_tid=13101]]], state=connected] 2024-04-24T00:55:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=13104], [object_number=135, tag_tid=29, entry_tid=13103], [object_number=134, tag_tid=29, entry_tid=13102], [object_number=133, tag_tid=29, entry_tid=13101]]], state=connected] 2024-04-24T00:55:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=132, tag_tid=29, entry_tid=13104' 2024-04-24T00:55:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=132, tag_tid=29, entry_tid=13104' 2024-04-24T00:55:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=13104' '!=' 'object_number=135, tag_tid=29, entry_tid=12935' 2024-04-24T00:55:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=13104' = 'object_number=135, tag_tid=29, entry_tid=12935' 2024-04-24T00:55:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=132, tag_tid=29, entry_tid=13104' 2024-04-24T00:55:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=13104' 2024-04-24T00:55:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=13104' 2024-04-24T00:55:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13104 2024-04-24T00:55:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 13104 2024-04-24T00:55:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=13248], [object_number=135, tag_tid=29, entry_tid=13247], [object_number=134, tag_tid=29, entry_tid=13246], [object_number=133, tag_tid=29, entry_tid=13245]]], state=connected] 2024-04-24T00:55:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 33 2024-04-24T00:55:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=132, tag_tid=29, entry_tid=13248], [object_number=135, tag_tid=29, entry_tid=13247], [object_number=134, tag_tid=29, entry_tid=13246], [object_number=133, tag_tid=29, entry_tid=13245]]], state=connected] 2024-04-24T00:55:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=132, tag_tid=29, entry_tid=13248' 2024-04-24T00:55:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=132, tag_tid=29, entry_tid=13248' 2024-04-24T00:55:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=13248' '!=' 'object_number=132, tag_tid=29, entry_tid=13104' 2024-04-24T00:55:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=132, tag_tid=29, entry_tid=13248' = 'object_number=132, tag_tid=29, entry_tid=13104' 2024-04-24T00:55:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=132, tag_tid=29, entry_tid=13248' 2024-04-24T00:55:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=13248' 2024-04-24T00:55:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=132, tag_tid=29, entry_tid=13248' 2024-04-24T00:55:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13248 2024-04-24T00:55:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 13248 2024-04-24T00:55:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=13417], [object_number=136, tag_tid=29, entry_tid=13416], [object_number=139, tag_tid=29, entry_tid=13415], [object_number=138, tag_tid=29, entry_tid=13414]]], state=connected] 2024-04-24T00:55:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=13417], [object_number=136, tag_tid=29, entry_tid=13416], [object_number=139, tag_tid=29, entry_tid=13415], [object_number=138, tag_tid=29, entry_tid=13414]]], state=connected] 2024-04-24T00:55:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=13417' 2024-04-24T00:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=13417' 2024-04-24T00:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=13417' '!=' 'object_number=132, tag_tid=29, entry_tid=13248' 2024-04-24T00:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=13417' = 'object_number=132, tag_tid=29, entry_tid=13248' 2024-04-24T00:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=13417' 2024-04-24T00:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=13417' 2024-04-24T00:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=13417' 2024-04-24T00:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13417 2024-04-24T00:55:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 13417 2024-04-24T00:55:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=13586], [object_number=137, tag_tid=29, entry_tid=13585], [object_number=136, tag_tid=29, entry_tid=13584], [object_number=139, tag_tid=29, entry_tid=13583]]], state=connected] 2024-04-24T00:55:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=13586], [object_number=137, tag_tid=29, entry_tid=13585], [object_number=136, tag_tid=29, entry_tid=13584], [object_number=139, tag_tid=29, entry_tid=13583]]], state=connected] 2024-04-24T00:55:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=13586' 2024-04-24T00:55:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=13586' 2024-04-24T00:55:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=13586' '!=' 'object_number=137, tag_tid=29, entry_tid=13417' 2024-04-24T00:55:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=13586' = 'object_number=137, tag_tid=29, entry_tid=13417' 2024-04-24T00:55:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=13586' 2024-04-24T00:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=13586' 2024-04-24T00:55:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=13586' 2024-04-24T00:55:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13586 2024-04-24T00:55:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 13586 2024-04-24T00:55:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=13827], [object_number=138, tag_tid=29, entry_tid=13826], [object_number=137, tag_tid=29, entry_tid=13825], [object_number=136, tag_tid=29, entry_tid=13824]]], state=connected] 2024-04-24T00:55:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=13827], [object_number=138, tag_tid=29, entry_tid=13826], [object_number=137, tag_tid=29, entry_tid=13825], [object_number=136, tag_tid=29, entry_tid=13824]]], state=connected] 2024-04-24T00:55:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=139, tag_tid=29, entry_tid=13827' 2024-04-24T00:55:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=139, tag_tid=29, entry_tid=13827' 2024-04-24T00:55:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=13827' '!=' 'object_number=138, tag_tid=29, entry_tid=13586' 2024-04-24T00:55:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=13827' = 'object_number=138, tag_tid=29, entry_tid=13586' 2024-04-24T00:55:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=139, tag_tid=29, entry_tid=13827' 2024-04-24T00:55:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=13827' 2024-04-24T00:55:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=13827' 2024-04-24T00:55:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13827 2024-04-24T00:55:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 13827 2024-04-24T00:55:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:20.918 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T00:55:20.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T00:55:20.919 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-24T00:55:20.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T00:55:20.918+0000 7fb3b125a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 81231) UID: 1000 2024-04-24T00:55:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=13976], [object_number=139, tag_tid=29, entry_tid=13975], [object_number=138, tag_tid=29, entry_tid=13974], [object_number=137, tag_tid=29, entry_tid=13973]]], state=connected] 2024-04-24T00:55:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=13976], [object_number=139, tag_tid=29, entry_tid=13975], [object_number=138, tag_tid=29, entry_tid=13974], [object_number=137, tag_tid=29, entry_tid=13973]]], state=connected] 2024-04-24T00:55:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=13976' 2024-04-24T00:55:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=13976' 2024-04-24T00:55:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=13976' '!=' 'object_number=139, tag_tid=29, entry_tid=13827' 2024-04-24T00:55:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=13976' = 'object_number=139, tag_tid=29, entry_tid=13827' 2024-04-24T00:55:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=13976' 2024-04-24T00:55:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=13976' 2024-04-24T00:55:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=13976' 2024-04-24T00:55:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=13976 2024-04-24T00:55:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 13976 2024-04-24T00:55:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=14119], [object_number=138, tag_tid=29, entry_tid=14118], [object_number=137, tag_tid=29, entry_tid=14117], [object_number=136, tag_tid=29, entry_tid=14116]]], state=connected] 2024-04-24T00:55:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=14119], [object_number=138, tag_tid=29, entry_tid=14118], [object_number=137, tag_tid=29, entry_tid=14117], [object_number=136, tag_tid=29, entry_tid=14116]]], state=connected] 2024-04-24T00:55:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=139, tag_tid=29, entry_tid=14119' 2024-04-24T00:55:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=139, tag_tid=29, entry_tid=14119' 2024-04-24T00:55:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=14119' '!=' 'object_number=136, tag_tid=29, entry_tid=13976' 2024-04-24T00:55:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=14119' = 'object_number=136, tag_tid=29, entry_tid=13976' 2024-04-24T00:55:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=139, tag_tid=29, entry_tid=14119' 2024-04-24T00:55:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=14119' 2024-04-24T00:55:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=14119' 2024-04-24T00:55:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14119 2024-04-24T00:55:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 14119 2024-04-24T00:55:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=14268], [object_number=139, tag_tid=29, entry_tid=14267], [object_number=138, tag_tid=29, entry_tid=14266], [object_number=137, tag_tid=29, entry_tid=14265]]], state=connected] 2024-04-24T00:55:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=14268], [object_number=139, tag_tid=29, entry_tid=14267], [object_number=138, tag_tid=29, entry_tid=14266], [object_number=137, tag_tid=29, entry_tid=14265]]], state=connected] 2024-04-24T00:55:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=14268' 2024-04-24T00:55:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=14268' 2024-04-24T00:55:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=14268' '!=' 'object_number=139, tag_tid=29, entry_tid=14119' 2024-04-24T00:55:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=14268' = 'object_number=139, tag_tid=29, entry_tid=14119' 2024-04-24T00:55:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=14268' 2024-04-24T00:55:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=14268' 2024-04-24T00:55:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=14268' 2024-04-24T00:55:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14268 2024-04-24T00:55:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 14268 2024-04-24T00:55:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=14407], [object_number=138, tag_tid=29, entry_tid=14406], [object_number=137, tag_tid=29, entry_tid=14405], [object_number=136, tag_tid=29, entry_tid=14404]]], state=connected] 2024-04-24T00:55:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=14407], [object_number=138, tag_tid=29, entry_tid=14406], [object_number=137, tag_tid=29, entry_tid=14405], [object_number=136, tag_tid=29, entry_tid=14404]]], state=connected] 2024-04-24T00:55:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=139, tag_tid=29, entry_tid=14407' 2024-04-24T00:55:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=139, tag_tid=29, entry_tid=14407' 2024-04-24T00:55:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=14407' '!=' 'object_number=136, tag_tid=29, entry_tid=14268' 2024-04-24T00:55:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=14407' = 'object_number=136, tag_tid=29, entry_tid=14268' 2024-04-24T00:55:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=139, tag_tid=29, entry_tid=14407' 2024-04-24T00:55:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=14407' 2024-04-24T00:55:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=14407' 2024-04-24T00:55:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14407 2024-04-24T00:55:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 14407 2024-04-24T00:55:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:23.920 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T00:55:23.920 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:55:23.920 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T00:55:23.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T00:55:23.921 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T00:55:23.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T00:55:23.923 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-04-24T00:55:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=14555], [object_number=138, tag_tid=29, entry_tid=14554], [object_number=137, tag_tid=29, entry_tid=14553], [object_number=136, tag_tid=29, entry_tid=14552]]], state=connected] 2024-04-24T00:55:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=14555], [object_number=138, tag_tid=29, entry_tid=14554], [object_number=137, tag_tid=29, entry_tid=14553], [object_number=136, tag_tid=29, entry_tid=14552]]], state=connected] 2024-04-24T00:55:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=139, tag_tid=29, entry_tid=14555' 2024-04-24T00:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=139, tag_tid=29, entry_tid=14555' 2024-04-24T00:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=14555' '!=' 'object_number=139, tag_tid=29, entry_tid=14407' 2024-04-24T00:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=14555' = 'object_number=139, tag_tid=29, entry_tid=14407' 2024-04-24T00:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=139, tag_tid=29, entry_tid=14555' 2024-04-24T00:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=14555' 2024-04-24T00:55:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=14555' 2024-04-24T00:55:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14555 2024-04-24T00:55:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 14555 2024-04-24T00:55:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=14701], [object_number=136, tag_tid=29, entry_tid=14700], [object_number=139, tag_tid=29, entry_tid=14699], [object_number=138, tag_tid=29, entry_tid=14698]]], state=connected] 2024-04-24T00:55:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=14701], [object_number=136, tag_tid=29, entry_tid=14700], [object_number=139, tag_tid=29, entry_tid=14699], [object_number=138, tag_tid=29, entry_tid=14698]]], state=connected] 2024-04-24T00:55:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=14701' 2024-04-24T00:55:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=14701' 2024-04-24T00:55:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=14701' '!=' 'object_number=139, tag_tid=29, entry_tid=14555' 2024-04-24T00:55:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=14701' = 'object_number=139, tag_tid=29, entry_tid=14555' 2024-04-24T00:55:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=14701' 2024-04-24T00:55:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=14701' 2024-04-24T00:55:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=14701' 2024-04-24T00:55:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14701 2024-04-24T00:55:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 14701 2024-04-24T00:55:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=14856], [object_number=139, tag_tid=29, entry_tid=14855], [object_number=138, tag_tid=29, entry_tid=14854], [object_number=137, tag_tid=29, entry_tid=14853]]], state=connected] 2024-04-24T00:55:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=14856], [object_number=139, tag_tid=29, entry_tid=14855], [object_number=138, tag_tid=29, entry_tid=14854], [object_number=137, tag_tid=29, entry_tid=14853]]], state=connected] 2024-04-24T00:55:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=14856' 2024-04-24T00:55:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=14856' 2024-04-24T00:55:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=14856' '!=' 'object_number=137, tag_tid=29, entry_tid=14701' 2024-04-24T00:55:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=14856' = 'object_number=137, tag_tid=29, entry_tid=14701' 2024-04-24T00:55:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=14856' 2024-04-24T00:55:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=14856' 2024-04-24T00:55:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=14856' 2024-04-24T00:55:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=14856 2024-04-24T00:55:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 14856 2024-04-24T00:55:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=15011], [object_number=138, tag_tid=29, entry_tid=15010], [object_number=137, tag_tid=29, entry_tid=15009], [object_number=136, tag_tid=29, entry_tid=15008]]], state=connected] 2024-04-24T00:55:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=15011], [object_number=138, tag_tid=29, entry_tid=15010], [object_number=137, tag_tid=29, entry_tid=15009], [object_number=136, tag_tid=29, entry_tid=15008]]], state=connected] 2024-04-24T00:55:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=139, tag_tid=29, entry_tid=15011' 2024-04-24T00:55:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=139, tag_tid=29, entry_tid=15011' 2024-04-24T00:55:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=15011' '!=' 'object_number=136, tag_tid=29, entry_tid=14856' 2024-04-24T00:55:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=15011' = 'object_number=136, tag_tid=29, entry_tid=14856' 2024-04-24T00:55:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=139, tag_tid=29, entry_tid=15011' 2024-04-24T00:55:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=15011' 2024-04-24T00:55:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:27.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:27.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:27.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=15011' 2024-04-24T00:55:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15011 2024-04-24T00:55:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 15011 2024-04-24T00:55:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=15166], [object_number=137, tag_tid=29, entry_tid=15165], [object_number=136, tag_tid=29, entry_tid=15164], [object_number=139, tag_tid=29, entry_tid=15163]]], state=connected] 2024-04-24T00:55:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=15166], [object_number=137, tag_tid=29, entry_tid=15165], [object_number=136, tag_tid=29, entry_tid=15164], [object_number=139, tag_tid=29, entry_tid=15163]]], state=connected] 2024-04-24T00:55:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=15166' 2024-04-24T00:55:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=15166' 2024-04-24T00:55:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=15166' '!=' 'object_number=139, tag_tid=29, entry_tid=15011' 2024-04-24T00:55:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=15166' = 'object_number=139, tag_tid=29, entry_tid=15011' 2024-04-24T00:55:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=15166' 2024-04-24T00:55:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=15166' 2024-04-24T00:55:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=15166' 2024-04-24T00:55:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15166 2024-04-24T00:55:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 15166 2024-04-24T00:55:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=15318], [object_number=137, tag_tid=29, entry_tid=15317], [object_number=136, tag_tid=29, entry_tid=15316], [object_number=139, tag_tid=29, entry_tid=15315]]], state=connected] 2024-04-24T00:55:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=15318], [object_number=137, tag_tid=29, entry_tid=15317], [object_number=136, tag_tid=29, entry_tid=15316], [object_number=139, tag_tid=29, entry_tid=15315]]], state=connected] 2024-04-24T00:55:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=15318' 2024-04-24T00:55:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=15318' 2024-04-24T00:55:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=15318' '!=' 'object_number=138, tag_tid=29, entry_tid=15166' 2024-04-24T00:55:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=15318' = 'object_number=138, tag_tid=29, entry_tid=15166' 2024-04-24T00:55:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=15318' 2024-04-24T00:55:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=15318' 2024-04-24T00:55:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=15318' 2024-04-24T00:55:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15318 2024-04-24T00:55:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 15318 2024-04-24T00:55:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=15470], [object_number=137, tag_tid=29, entry_tid=15469], [object_number=136, tag_tid=29, entry_tid=15468], [object_number=139, tag_tid=29, entry_tid=15467]]], state=connected] 2024-04-24T00:55:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=15470], [object_number=137, tag_tid=29, entry_tid=15469], [object_number=136, tag_tid=29, entry_tid=15468], [object_number=139, tag_tid=29, entry_tid=15467]]], state=connected] 2024-04-24T00:55:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=15470' 2024-04-24T00:55:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=15470' 2024-04-24T00:55:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=15470' '!=' 'object_number=138, tag_tid=29, entry_tid=15318' 2024-04-24T00:55:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=15470' = 'object_number=138, tag_tid=29, entry_tid=15318' 2024-04-24T00:55:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=15470' 2024-04-24T00:55:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=15470' 2024-04-24T00:55:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=15470' 2024-04-24T00:55:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15470 2024-04-24T00:55:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 15470 2024-04-24T00:55:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=15613], [object_number=136, tag_tid=29, entry_tid=15612], [object_number=139, tag_tid=29, entry_tid=15611], [object_number=138, tag_tid=29, entry_tid=15610]]], state=connected] 2024-04-24T00:55:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=15613], [object_number=136, tag_tid=29, entry_tid=15612], [object_number=139, tag_tid=29, entry_tid=15611], [object_number=138, tag_tid=29, entry_tid=15610]]], state=connected] 2024-04-24T00:55:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=15613' 2024-04-24T00:55:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=15613' 2024-04-24T00:55:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=15613' '!=' 'object_number=138, tag_tid=29, entry_tid=15470' 2024-04-24T00:55:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=15613' = 'object_number=138, tag_tid=29, entry_tid=15470' 2024-04-24T00:55:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=15613' 2024-04-24T00:55:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=15613' 2024-04-24T00:55:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=15613' 2024-04-24T00:55:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15613 2024-04-24T00:55:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 15613 2024-04-24T00:55:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=15774], [object_number=137, tag_tid=29, entry_tid=15773], [object_number=136, tag_tid=29, entry_tid=15772], [object_number=139, tag_tid=29, entry_tid=15771]]], state=connected] 2024-04-24T00:55:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=15774], [object_number=137, tag_tid=29, entry_tid=15773], [object_number=136, tag_tid=29, entry_tid=15772], [object_number=139, tag_tid=29, entry_tid=15771]]], state=connected] 2024-04-24T00:55:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=15774' 2024-04-24T00:55:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=15774' 2024-04-24T00:55:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=15774' '!=' 'object_number=137, tag_tid=29, entry_tid=15613' 2024-04-24T00:55:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=15774' = 'object_number=137, tag_tid=29, entry_tid=15613' 2024-04-24T00:55:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=15774' 2024-04-24T00:55:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=15774' 2024-04-24T00:55:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=15774' 2024-04-24T00:55:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15774 2024-04-24T00:55:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 15774 2024-04-24T00:55:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=15933], [object_number=136, tag_tid=29, entry_tid=15932], [object_number=139, tag_tid=29, entry_tid=15931], [object_number=138, tag_tid=29, entry_tid=15930]]], state=connected] 2024-04-24T00:55:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=15933], [object_number=136, tag_tid=29, entry_tid=15932], [object_number=139, tag_tid=29, entry_tid=15931], [object_number=138, tag_tid=29, entry_tid=15930]]], state=connected] 2024-04-24T00:55:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=15933' 2024-04-24T00:55:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=15933' 2024-04-24T00:55:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=15933' '!=' 'object_number=138, tag_tid=29, entry_tid=15774' 2024-04-24T00:55:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=15933' = 'object_number=138, tag_tid=29, entry_tid=15774' 2024-04-24T00:55:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=15933' 2024-04-24T00:55:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=15933' 2024-04-24T00:55:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=15933' 2024-04-24T00:55:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15933 2024-04-24T00:55:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 15933 2024-04-24T00:55:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=16076], [object_number=139, tag_tid=29, entry_tid=16075], [object_number=138, tag_tid=29, entry_tid=16074], [object_number=137, tag_tid=29, entry_tid=16073]]], state=connected] 2024-04-24T00:55:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=16076], [object_number=139, tag_tid=29, entry_tid=16075], [object_number=138, tag_tid=29, entry_tid=16074], [object_number=137, tag_tid=29, entry_tid=16073]]], state=connected] 2024-04-24T00:55:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=16076' 2024-04-24T00:55:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=16076' 2024-04-24T00:55:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=16076' '!=' 'object_number=137, tag_tid=29, entry_tid=15933' 2024-04-24T00:55:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=16076' = 'object_number=137, tag_tid=29, entry_tid=15933' 2024-04-24T00:55:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=16076' 2024-04-24T00:55:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=16076' 2024-04-24T00:55:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=16076' 2024-04-24T00:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16076 2024-04-24T00:55:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 16076 2024-04-24T00:55:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=16230], [object_number=137, tag_tid=29, entry_tid=16229], [object_number=136, tag_tid=29, entry_tid=16228], [object_number=139, tag_tid=29, entry_tid=16227]]], state=connected] 2024-04-24T00:55:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=16230], [object_number=137, tag_tid=29, entry_tid=16229], [object_number=136, tag_tid=29, entry_tid=16228], [object_number=139, tag_tid=29, entry_tid=16227]]], state=connected] 2024-04-24T00:55:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=16230' 2024-04-24T00:55:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=16230' 2024-04-24T00:55:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=16230' '!=' 'object_number=136, tag_tid=29, entry_tid=16076' 2024-04-24T00:55:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=16230' = 'object_number=136, tag_tid=29, entry_tid=16076' 2024-04-24T00:55:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=16230' 2024-04-24T00:55:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=16230' 2024-04-24T00:55:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=16230' 2024-04-24T00:55:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16230 2024-04-24T00:55:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 16230 2024-04-24T00:55:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=16272], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270], [object_number=137, tag_tid=29, entry_tid=16269]]], state=connected] 2024-04-24T00:55:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=16272], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270], [object_number=137, tag_tid=29, entry_tid=16269]]], state=connected] 2024-04-24T00:55:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=16272' 2024-04-24T00:55:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=16272' 2024-04-24T00:55:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=16272' '!=' 'object_number=138, tag_tid=29, entry_tid=16230' 2024-04-24T00:55:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=16272' = 'object_number=138, tag_tid=29, entry_tid=16230' 2024-04-24T00:55:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=16272' 2024-04-24T00:55:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=30 2024-04-24T00:55:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=16272' 2024-04-24T00:55:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=29 2024-04-24T00:55:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=16272' 2024-04-24T00:55:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=16272 2024-04-24T00:55:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 30 = 29 -a 1 -le 16272 2024-04-24T00:55:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:36.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-04-24T00:55:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10 2024-04-24T00:55:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:55:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:55:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:55:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-04-24T00:55:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:55:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 19 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:53:50 2024 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:55:10 2024 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:55:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:55:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:55:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:55:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:55:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap9-clean ']' 2024-04-24T00:55:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-24T00:55:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9-clean 2024-04-24T00:55:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:55:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:38.518 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:55:38.518+0000 7efd4dcc6640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:55:38.809 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3264783823737022) > weight (0.25) 2024-04-24T00:55:38.809 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T00:55:38.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T00:55:38.809 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-24T00:55:38.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T00:55:38.809+0000 7f0b4ab3d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 83784) UID: 1000 2024-04-24T00:55:38.940 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:55:38.942+0000 7fbc2ca22640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:55:38.941 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:55:38.942+0000 7fbc28a1a640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:55:38.945 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:55:38.946+0000 7fbc28219640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-04-24T00:55:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap10 2024-04-24T00:55:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10 2024-04-24T00:55:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:55:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:41.074 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:55:41.074+0000 7efd4dcc6640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:55:41.182 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:55:41.182+0000 7fbc2ca22640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:55:41.376 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:55:41.378+0000 7fbc2f227640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:55:41.376 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:55:41.378+0000 7fbc2b21f640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-04-24T00:55:41.378 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:55:41.382+0000 7fbc28a1a640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:55:41.486 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:55:41.486+0000 7efd4c4c3640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-04-24T00:55:41.487 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:55:41.490+0000 7efd534d1640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 8 2024-04-24T00:55:41.487 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:55:41.490+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-04-24T00:55:41.491 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:55:41.494+0000 7efd52cd0640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 8 2024-04-24T00:55:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap10-clean 2024-04-24T00:55:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-04-24T00:55:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:55:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10-clean 2024-04-24T00:55:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T00:55:42.317 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:55:42.317+0000 7ff578350640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2024-04-24T00:55:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T00:55:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T00:55:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T00:55:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:55:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:55:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T00:55:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T00:55:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T00:55:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T00:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T00:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T00:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T00:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T00:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T00:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T00:55:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T00:55:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T00:55:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T00:55:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:55:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T00:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:55:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:55:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:55:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:55:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:55:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T00:55:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T00:55:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T00:55:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:55:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T00:55:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T00:55:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T00:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T00:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T00:55:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:55:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T00:55:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T00:55:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T00:55:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T00:55:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T00:55:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T00:55:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T00:55:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T00:55:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T00:55:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T00:55:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T00:55:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T00:55:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T00:55:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T00:55:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T00:55:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-24T00:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T00:55:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:55:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T00:55:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T00:55:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=31, entry_tid=1' -a 'object_number=137, tag_tid=31, entry_tid=1' = 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=30, entry_tid=1' '!=' '' 2024-04-24T00:55:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T00:55:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=30, entry_tid=1' = '' 2024-04-24T00:55:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=31 2024-04-24T00:55:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T00:55:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=30 2024-04-24T00:55:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T00:55:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T00:55:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T00:55:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=1 2024-04-24T00:55:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 31 = 30 -a 1 -le 1 2024-04-24T00:55:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T00:55:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:55:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:46.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=31, entry_tid=1' -a 'object_number=137, tag_tid=31, entry_tid=1' = 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=30, entry_tid=1' '!=' 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T00:55:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=31, entry_tid=1' -a 'object_number=137, tag_tid=31, entry_tid=1' = 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=30, entry_tid=1' '!=' 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T00:55:47.810 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T00:55:47.811 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:55:47.811 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T00:55:47.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T00:55:47.811 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T00:55:47.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T00:55:47.814 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-04-24T00:55:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=31, entry_tid=1' -a 'object_number=137, tag_tid=31, entry_tid=1' = 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=30, entry_tid=1' '!=' 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=31, entry_tid=1' -a 'object_number=137, tag_tid=31, entry_tid=1' = 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=137, tag_tid=30, entry_tid=1' '!=' 'object_number=137, tag_tid=30, entry_tid=1' 2024-04-24T00:55:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T00:55:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T00:55:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror test 2024-04-24T00:55:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T00:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n test ']' 2024-04-24T00:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-24T00:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-24T00:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T00:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T00:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T00:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T00:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T00:55:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T00:55:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-24T00:55:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T00:55:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T00:55:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T00:55:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T00:55:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T00:55:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T00:55:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T00:55:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T00:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T00:55:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T00:55:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-24T00:55:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-24T00:55:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T00:55:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T00:55:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T00:55:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T00:55:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T00:55:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T00:55:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T00:55:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T00:55:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T00:55:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T00:55:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-24T00:55:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2024-04-24T00:55:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T00:55:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T00:55:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T00:55:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T00:55:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T00:55:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T00:55:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T00:55:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T00:55:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T00:55:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T00:55:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T00:55:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2024-04-24T00:55:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T00:55:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T00:55:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T00:55:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T00:55:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test 2024-04-24T00:55:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T00:55:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:55:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T00:55:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T00:55:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T00:55:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T00:55:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T00:55:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T00:55:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T00:55:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T00:55:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T00:55:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T00:55:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T00:55:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:55:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T00:55:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T00:55:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T00:55:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T00:55:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T00:55:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T00:55:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T00:55:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T00:55:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T00:55:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T00:55:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T00:55:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T00:55:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T00:55:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T00:55:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test 2024-04-24T00:55:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-24T00:55:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-24T00:55:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T00:55:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-24T00:55:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-test.status 2024-04-24T00:55:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T00:55:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-24T00:55:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 34 2024-04-24T00:55:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 34 2024-04-24T00:55:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T00:55:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=16271], [object_number=138, tag_tid=29, entry_tid=16270]]], state=connected] 2024-04-24T00:55:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.status 2024-04-24T00:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=137, tag_tid=31, entry_tid=1' -a 'object_number=137, tag_tid=31, entry_tid=1' = 'object_number=137, tag_tid=31, entry_tid=1' 2024-04-24T00:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-04-24T00:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10-clean 2024-04-24T00:55:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:55:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T00:55:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T00:55:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-04-24T00:55:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T00:55:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T00:55:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T00:55:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 20 2024-04-24T00:55:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10ae357a5e51 2024-04-24T00:55:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10ae357a5e51 2024-04-24T00:55:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T00:55:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:39:28 2024 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:55:38 2024 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:55:10 2024 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10ae357a5e51 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b94c2353-363b-4139-b9b9-403e9094bfc3 2024-04-24T00:55:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T00:55:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 10 2024-04-24T00:55:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:55:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1 2024-04-24T00:55:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap1 2024-04-24T00:55:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1 2024-04-24T00:55:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:55:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1-clean 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1-clean 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:56.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:55:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2 2024-04-24T00:55:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap2 2024-04-24T00:55:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2 2024-04-24T00:55:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:55:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2-clean 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2-clean 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:55:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:55:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:55:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:00.222 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:00.221+0000 7efd524cf640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:00.223 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:00.221+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap3 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:02.452 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:02.453+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3-clean 2024-04-24T00:56:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-24T00:56:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3-clean 2024-04-24T00:56:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:04.657 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:04.657+0000 7efd4ecc8640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:04.763 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:04.761+0000 7efd4fcca640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:04.763 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:04.761+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:05.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4 2024-04-24T00:56:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap4 2024-04-24T00:56:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4 2024-04-24T00:56:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:06.957 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:06.957+0000 7efd4ccc4640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:06.958 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:06.957+0000 7efd504cb640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:06.959 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:06.961+0000 7efd524cf640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:06.959 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:06.961+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4-clean 2024-04-24T00:56:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-24T00:56:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4-clean 2024-04-24T00:56:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:08.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:08.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:08.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:09.221 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:09.221+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:09.221 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:09.221+0000 7efd4c4c3640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:09.221 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:09.221+0000 7efd524cf640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:09.222 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:09.221+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T00:56:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5 2024-04-24T00:56:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap5 2024-04-24T00:56:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5 2024-04-24T00:56:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:11.503 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:11.501+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:11.503 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:11.505+0000 7efd4d4c5640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:11.503 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:11.505+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:11.713 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:11.713+0000 7fbc28219640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5-clean 2024-04-24T00:56:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-24T00:56:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5-clean 2024-04-24T00:56:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:13.790 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:13.789+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:13.791 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:13.793+0000 7efd4f4c9640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:14.093 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:14.093+0000 7fbc2aa1e640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:14.096 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:14.097+0000 7fbc2c221640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:14.215 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:14.217+0000 7fbc2ea26640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:14.216 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:14.217+0000 7fbc28219640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap6 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:16.246 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:16.245+0000 7efd534d1640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:16.247 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:16.245+0000 7efd4d4c5640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:16.248 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:16.249+0000 7efd4ecc8640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:16.694 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:16.693+0000 7efd52cd0640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:16.694 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:16.693+0000 7efd514cd640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:16.695 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:16.697+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6-clean 2024-04-24T00:56:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-24T00:56:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6-clean 2024-04-24T00:56:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:18.480 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:18.481+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7 2024-04-24T00:56:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap7 2024-04-24T00:56:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7 2024-04-24T00:56:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:21.588 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:21.589+0000 7efd534d1640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:21.589 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:21.589+0000 7efd4ecc8640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:21.685 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:21.685+0000 7efd51cce640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7-clean 2024-04-24T00:56:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-24T00:56:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7-clean 2024-04-24T00:56:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:23.449 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:23.449+0000 7efd524cf640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:23.450 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:23.449+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:23.555 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:23.553+0000 7efd4ecc8640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:23.768 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:23.769+0000 7fbc28219640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:23.768 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:23.769+0000 7fbc2aa1e640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:23.770 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:23.769+0000 7fbc2f227640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-04-24T00:56:23.770 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:23.769+0000 7fbc2ca22640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-04-24T00:56:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap8 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:25.820 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:25.821+0000 7efd4ccc4640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:25.821 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:25.821+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:26.266 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:26.265+0000 7fbc2b21f640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-04-24T00:56:26.267 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:26.269+0000 7fbc28a1a640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-04-24T00:56:26.388 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:26.389+0000 7efd4fcca640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8-clean 2024-04-24T00:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-24T00:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8-clean 2024-04-24T00:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:27.420 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:56:27.420+0000 7fbdbe182640 -1 reset not still connected to 0x5622d0177520 2024-04-24T00:56:27.420 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:56:27.420+0000 7fbdbe182640 -1 reset not still connected to 0x5622d01fc9c0 2024-04-24T00:56:27.420 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:56:27.420+0000 7fbdbe182640 -1 reset not still connected to 0x5622d02695f0 2024-04-24T00:56:27.429 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:56:27.428+0000 7fbdbe182640 -1 reset not still connected to 0x5622d164af70 2024-04-24T00:56:27.429 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:56:27.428+0000 7fbdbe182640 -1 reset not still connected to 0x5622d5355110 2024-04-24T00:56:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:27.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:28.955 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:28.956+0000 7efd4d4c5640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T00:56:28.962 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:28.960+0000 7efd4fcca640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-04-24T00:56:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap9 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:31.339 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:31.336+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:31.339 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:31.336+0000 7efd534d1640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:31.439 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:31.440+0000 7efd4c4c3640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:31.873 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:31.872+0000 7efd4ccc4640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:31.873 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:31.872+0000 7efd52cd0640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:31.874 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:31.872+0000 7efd4d4c5640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:31.877 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:31.876+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 10 2024-04-24T00:56:32.081 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:32.080+0000 7efd4f4c9640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9-clean 2024-04-24T00:56:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-24T00:56:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9-clean 2024-04-24T00:56:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:33.792 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:33.792+0000 7efd4f4c9640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T00:56:33.794 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:33.792+0000 7efd4ecc8640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T00:56:34.126 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:34.124+0000 7fbc28219640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:34.127 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:34.128+0000 7fbc2fa28640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:34.244 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:34.244+0000 7fbc28219640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:34.246 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:34.244+0000 7fbc2ca22640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:34.246 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:34.248+0000 7fbc2aa1e640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10 2024-04-24T00:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap10 2024-04-24T00:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10 2024-04-24T00:56:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:36.250 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:36.248+0000 7efd4ccc4640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:36.662 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:56:36.660+0000 7fbc2aa1e640 -1 bdev(0x556da7dbae00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-04-24T00:56:36.795 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:36.796+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T00:56:36.797 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:36.796+0000 7efd52cd0640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2024-04-24T00:56:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10-clean 2024-04-24T00:56:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap10-clean 2024-04-24T00:56:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10-clean 2024-04-24T00:56:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T00:56:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export 2024-04-24T00:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:38.431 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:38.432+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T00:56:38.680 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:38.680+0000 7efd4fcca640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T00:56:38.680 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:38.680+0000 7efd4d4c5640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-04-24T00:56:38.682 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:38.680+0000 7efd4ecc8640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-04-24T00:56:38.789 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:38.788+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:38.894 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:38.888+0000 7efd4dcc6640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T00:56:38.894 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:38.892+0000 7efd51cce640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T00:56:38.901 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:56:38.892+0000 7efd4c4c3640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T00:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T00:56:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-test.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-test.export 2024-04-24T00:56:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:56:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 10 2024-04-24T00:56:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1 2024-04-24T00:56:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-04-24T00:56:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap1 2024-04-24T00:56:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-04-24T00:56:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1-clean 2024-04-24T00:56:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-04-24T00:56:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap1-clean 2024-04-24T00:56:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-04-24T00:56:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2 2024-04-24T00:56:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-04-24T00:56:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap2 2024-04-24T00:56:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-04-24T00:56:41.841 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:41.844+0000 7ff578b51640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2024-04-24T00:56:41.843 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:41.844+0000 7ff574348640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2024-04-24T00:56:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2-clean 2024-04-24T00:56:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-04-24T00:56:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap2-clean 2024-04-24T00:56:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-04-24T00:56:42.837 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:42.836+0000 7ff578350640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2024-04-24T00:56:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3 2024-04-24T00:56:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-04-24T00:56:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap3 2024-04-24T00:56:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-04-24T00:56:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3-clean 2024-04-24T00:56:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-04-24T00:56:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap3-clean 2024-04-24T00:56:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-04-24T00:56:44.849 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:44.852+0000 7ff574348640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2024-04-24T00:56:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4 2024-04-24T00:56:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-04-24T00:56:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap4 2024-04-24T00:56:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-04-24T00:56:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4-clean 2024-04-24T00:56:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-04-24T00:56:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap4-clean 2024-04-24T00:56:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-04-24T00:56:46.876 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:46.876+0000 7ff578350640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 6 2024-04-24T00:56:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5 2024-04-24T00:56:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-04-24T00:56:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap5 2024-04-24T00:56:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-04-24T00:56:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5-clean 2024-04-24T00:56:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-04-24T00:56:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap5-clean 2024-04-24T00:56:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-04-24T00:56:48.916 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:48.915+0000 7ff574348640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 8 2024-04-24T00:56:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6 2024-04-24T00:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-04-24T00:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap6 2024-04-24T00:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-04-24T00:56:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6-clean 2024-04-24T00:56:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-04-24T00:56:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap6-clean 2024-04-24T00:56:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-04-24T00:56:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7 2024-04-24T00:56:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-04-24T00:56:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap7 2024-04-24T00:56:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-04-24T00:56:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7-clean 2024-04-24T00:56:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-04-24T00:56:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap7-clean 2024-04-24T00:56:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-04-24T00:56:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8 2024-04-24T00:56:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-04-24T00:56:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap8 2024-04-24T00:56:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-04-24T00:56:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8-clean 2024-04-24T00:56:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-04-24T00:56:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap8-clean 2024-04-24T00:56:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-04-24T00:56:54.945 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:54.943+0000 7ff578350640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2024-04-24T00:56:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9 2024-04-24T00:56:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-04-24T00:56:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap9 2024-04-24T00:56:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-04-24T00:56:55.995 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:55.995+0000 7ff578350640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 19 2024-04-24T00:56:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9-clean 2024-04-24T00:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-04-24T00:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap9-clean 2024-04-24T00:56:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-04-24T00:56:56.966 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:56.967+0000 7ff574b49640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2024-04-24T00:56:56.968 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:56:56.967+0000 7ff574348640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 9 2024-04-24T00:56:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T00:56:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10 2024-04-24T00:56:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-04-24T00:56:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap10 2024-04-24T00:56:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-04-24T00:56:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10-clean 2024-04-24T00:56:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-04-24T00:56:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap10-clean 2024-04-24T00:56:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-04-24T00:56:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T00:56:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror test 2024-04-24T00:56:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T00:56:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T00:56:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror test 2024-04-24T00:56:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:56:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:56:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:56:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-04-24T00:56:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-04-24T00:57:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T00:57:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:57:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-04-24T00:57:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:57:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:57:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T00:57:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T00:57:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T00:57:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T00:57:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror test 2024-04-24T00:57:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:57:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:57:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:57:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-24T00:57:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T00:57:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-24T00:57:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T00:57:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T00:57:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror test 2024-04-24T00:57:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T00:57:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T00:57:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T00:57:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-24T00:57:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T00:57:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-24T00:57:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T00:57:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T00:57:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T00:57:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:57:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: create many images' 2024-04-24T00:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.aEE7yWfHIo/rbd-mirror.test.log 2024-04-24T00:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T00:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 00:57:00 TEST: create many images 2024-04-24T00:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 00:57:00 TEST: create many images 2024-04-24T00:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap 2024-04-24T00:57:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 50 2024-04-24T00:57:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:57:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T00:57:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 journal 128M 2024-04-24T00:57:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_1 128M 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:57:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:57:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-04-24T00:57:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:57:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:57:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:57:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:57:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:57:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_1 2024-04-24T00:57:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T00:57:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:57:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T00:57:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:57:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:57:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 20382720 bytes 2024-04-24T00:57:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 97027072 bytes 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 journal 128M 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T00:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_2 128M 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-04-24T00:57:06.597 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:57:06.595+0000 7fbdbe182640 -1 reset not still connected to 0x5622e1a30270 2024-04-24T00:57:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_2 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T00:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:57:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T00:57:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:57:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:57:07.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:57:07.747+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3380 finish: resending after timeout 2024-04-24T00:57:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 43305984 bytes 2024-04-24T00:57:10.924 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T00:57:10.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T00:57:10.925 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-24T00:57:10.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T00:57:10.927+0000 7f2f66379640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 92521) UID: 1000 2024-04-24T00:57:12.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:57:12.747+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3380 finish: resending after timeout 2024-04-24T00:57:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 116146176 bytes 2024-04-24T00:57:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 208263168 bytes 2024-04-24T00:57:19.926 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T00:57:19.927 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:57:19.927 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T00:57:19.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T00:57:19.927 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T00:57:19.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T00:57:19.930 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-04-24T00:57:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:57:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:57:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:57:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 journal 128M 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_3 128M 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T00:57:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:57:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:57:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:57:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-04-24T00:57:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:57:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_3 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:57:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T00:57:20.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:57:20.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:57:23.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:57:23.527+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3380 finish: resending after timeout 2024-04-24T00:57:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 46654464 bytes 2024-04-24T00:57:25.314 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:57:25.315+0000 7f6307765640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-04-24T00:57:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 85518336 bytes 2024-04-24T00:57:28.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:57:28.531+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3380 finish: resending after timeout 2024-04-24T00:57:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 146038784 bytes 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 journal 128M 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_4 128M 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T00:57:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:57:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:57:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:57:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-04-24T00:57:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:57:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:57:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:57:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:57:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:57:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_4 2024-04-24T00:57:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T00:57:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:57:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T00:57:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:57:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:57:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 17226752 bytes 2024-04-24T00:57:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 33906688 bytes 2024-04-24T00:57:33.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:57:33.530+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3380 finish: resending after timeout 2024-04-24T00:57:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 34524160 bytes 2024-04-24T00:57:36.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9146805560632006) > weight (0.25) 2024-04-24T00:57:36.816 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T00:57:36.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T00:57:36.816 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-24T00:57:36.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T00:57:36.818+0000 7fc7f3629640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 94509) UID: 1000 2024-04-24T00:57:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 118727680 bytes 2024-04-24T00:57:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 139516928 bytes 2024-04-24T00:57:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:57:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:57:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 journal 128M 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:57:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_5 128M 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_5 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T00:57:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:57:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T00:57:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:57:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:57:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 25269248 bytes 2024-04-24T00:57:41.817 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T00:57:41.817 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:57:41.818 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T00:57:41.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T00:57:41.818 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T00:57:41.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T00:57:41.820 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-04-24T00:57:45.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 68328448 bytes 2024-04-24T00:57:45.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:57:45.486+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3380 finish: resending after timeout 2024-04-24T00:57:47.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:57:47.090+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c2000 finish: resending after timeout 2024-04-24T00:57:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 152801280 bytes 2024-04-24T00:57:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:57:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:57:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 journal 128M 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_6 128M 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:57:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-04-24T00:57:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:57:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:57:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:57:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:57:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:57:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_6 2024-04-24T00:57:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:57:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:57:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T00:57:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:57:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T00:57:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:57:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:57:52.257 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:57:52.259+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaeddc70 2024-04-24T00:57:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 4856832 bytes 2024-04-24T00:57:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 25907200 bytes 2024-04-24T00:57:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 56426496 bytes 2024-04-24T00:57:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 69236736 bytes 2024-04-24T00:57:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 69822464 bytes 2024-04-24T00:57:58.040 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T00:57:58.038+0000 7f63163ce640 -1 reset not still connected to 0x55c52e0c3ba0 2024-04-24T00:58:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 135389184 bytes 2024-04-24T00:58:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:58:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:58:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 journal 128M 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_7 128M 2024-04-24T00:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_7 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T00:58:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:58:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T00:58:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:58:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:58:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 48497664 bytes 2024-04-24T00:58:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 72229888 bytes 2024-04-24T00:58:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 110343168 bytes 2024-04-24T00:58:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:58:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 journal 128M 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:58:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_8 128M 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_8 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T00:58:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:58:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T00:58:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:58:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:58:14.491 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:58:14.494+0000 7efd60937640 -1 reset not still connected to 0x559b03182dd0 2024-04-24T00:58:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 89486336 bytes 2024-04-24T00:58:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 90102784 bytes 2024-04-24T00:58:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 166350848 bytes 2024-04-24T00:58:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:58:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:58:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:58:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:58:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 journal 128M 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_9 128M 2024-04-24T00:58:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-04-24T00:58:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:58:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:58:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:58:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:58:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:58:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_9 2024-04-24T00:58:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T00:58:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:58:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T00:58:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:58:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:58:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 56987648 bytes 2024-04-24T00:58:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 81634304 bytes 2024-04-24T00:58:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 89902080 bytes 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 journal 128M 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_10 128M 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T00:58:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:58:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:58:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:58:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-04-24T00:58:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:58:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:58:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:58:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:58:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:58:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_10 2024-04-24T00:58:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T00:58:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:58:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T00:58:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:58:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:58:33.931 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T00:58:33.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T00:58:33.932 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T00:58:33.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T00:58:33.929+0000 7fe51444f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 96432) UID: 1000 2024-04-24T00:58:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 73554944 bytes 2024-04-24T00:58:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 145436672 bytes 2024-04-24T00:58:41.933 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T00:58:41.933 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T00:58:41.933 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T00:58:41.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T00:58:41.934 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T00:58:41.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T00:58:41.937 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-04-24T00:58:43.293 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:58:43.294+0000 7efd60937640 -1 reset not still connected to 0x559aee775ee0 2024-04-24T00:58:43.293 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:58:43.294+0000 7efd60937640 -1 reset not still connected to 0x559b1beb8270 2024-04-24T00:58:43.293 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:58:43.294+0000 7efd60937640 -1 reset not still connected to 0x559b1c5560d0 2024-04-24T00:58:43.294 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:58:43.294+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T00:58:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 179147776 bytes 2024-04-24T00:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_11 2024-04-24T00:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 journal 128M 2024-04-24T00:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_11 128M 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T00:58:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:58:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:58:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:58:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_11 2024-04-24T00:58:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:58:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_11 2024-04-24T00:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T00:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:58:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T00:58:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:58:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:58:44.821 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.373393683299177) > weight (0.25) 2024-04-24T00:58:44.821 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9018955540400225) > weight (0.25) 2024-04-24T00:58:44.821 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.48465401810452435) > weight (0.25) 2024-04-24T00:58:44.821 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4537878499738115) > weight (0.25) 2024-04-24T00:58:44.821 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-04-24T00:58:45.932 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:58:45.930+0000 7fbc3ce8e640 -1 reset not still connected to 0x556da9e32820 2024-04-24T00:58:45.932 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:58:45.930+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaedc270 2024-04-24T00:58:45.936 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:58:45.938+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaeddc70 2024-04-24T00:58:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 24779776 bytes 2024-04-24T00:58:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 38187008 bytes 2024-04-24T00:58:47.784 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:58:47.785+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa7900d0 2024-04-24T00:58:47.784 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:58:47.785+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa791520 2024-04-24T00:58:47.785 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:58:47.785+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db032fc70 2024-04-24T00:58:47.785 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:58:47.785+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db2616000 2024-04-24T00:58:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 59954176 bytes 2024-04-24T00:58:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 77064192 bytes 2024-04-24T00:58:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:58:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:58:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:58:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:58:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:58:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_12 2024-04-24T00:58:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 journal 128M 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_12 128M 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T00:58:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:58:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:58:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:58:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:58:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_12 2024-04-24T00:58:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:58:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:58:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:58:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:58:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:58:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_12 2024-04-24T00:58:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:58:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:58:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T00:58:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:58:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T00:58:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:58:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:58:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 47726592 bytes 2024-04-24T00:58:53.752 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T00:58:53.753+0000 7ff576b4d640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 11 2024-04-24T00:58:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 52350976 bytes 2024-04-24T00:58:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 76982272 bytes 2024-04-24T00:58:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 94938112 bytes 2024-04-24T00:58:59.817 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:58:59.817+0000 7fd951b0e640 -1 reset not still connected to 0x5636f4e50410 2024-04-24T00:58:59.967 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:58:59.969+0000 7efd60937640 -1 reset not still connected to 0x559afb40f040 2024-04-24T00:58:59.968 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:58:59.969+0000 7efd60937640 -1 reset not still connected to 0x559b03182dd0 2024-04-24T00:59:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 132427776 bytes 2024-04-24T00:59:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_13 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 journal 128M 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:59:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_13 128M 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_13 2024-04-24T00:59:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:59:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:59:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:59:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:59:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:59:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_13 2024-04-24T00:59:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T00:59:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:59:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T00:59:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:59:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:59:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 42250240 bytes 2024-04-24T00:59:06.044 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:06.041+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc31aba00 2024-04-24T00:59:06.045 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:06.041+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T00:59:07.732 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:59:07.733+0000 7fbdaf519640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-04-24T00:59:07.733 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T00:59:07.737+0000 7fbdad515640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-04-24T00:59:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 92437504 bytes 2024-04-24T00:59:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 112972800 bytes 2024-04-24T00:59:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_14 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 journal 128M 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_14 128M 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_14 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_14 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T00:59:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:59:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T00:59:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:59:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:59:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 23311360 bytes 2024-04-24T00:59:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 42161152 bytes 2024-04-24T00:59:16.785 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:16.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db032fba0 2024-04-24T00:59:16.786 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:16.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db5d3e410 2024-04-24T00:59:16.787 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:16.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb38680 2024-04-24T00:59:16.787 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:16.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb736c0 2024-04-24T00:59:16.787 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:16.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc31aba00 2024-04-24T00:59:16.787 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:16.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T00:59:16.787 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:16.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T00:59:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 75515904 bytes 2024-04-24T00:59:20.283 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:20.280+0000 7fbc3ce8e640 -1 reset not still connected to 0x556da9e32820 2024-04-24T00:59:20.283 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:20.280+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaedc270 2024-04-24T00:59:20.284 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T00:59:20.280+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaeddc70 2024-04-24T00:59:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 143673344 bytes 2024-04-24T00:59:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 167762944 bytes 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_15 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 journal 128M 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_15 128M 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:59:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:59:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:59:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_15 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_15 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T00:59:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:59:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T00:59:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:59:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:59:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 25638912 bytes 2024-04-24T00:59:26.272 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:59:26.268+0000 7fd951b0e640 -1 reset not still connected to 0x5636f57e8750 2024-04-24T00:59:27.477 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:27.476+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T00:59:27.477 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:27.476+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T00:59:28.137 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:59:28.136+0000 7fd951b0e640 -1 reset not still connected to 0x563700359930 2024-04-24T00:59:28.137 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:59:28.136+0000 7fd951b0e640 -1 reset not still connected to 0x56371a3b1e10 2024-04-24T00:59:28.137 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:59:28.136+0000 7fd951b0e640 -1 reset not still connected to 0x563724a401a0 2024-04-24T00:59:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 75575296 bytes 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_16 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 journal 128M 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_16 128M 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:59:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:59:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_16 2024-04-24T00:59:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_16 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T00:59:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:59:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T00:59:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:59:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:59:32.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:59:32.164+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573197bc0c0 finish: resending after timeout 2024-04-24T00:59:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 75976704 bytes 2024-04-24T00:59:37.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:59:37.164+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573197bc0c0 finish: resending after timeout 2024-04-24T00:59:37.559 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:37.556+0000 7efd60937640 -1 reset not still connected to 0x559aeb2892b0 2024-04-24T00:59:37.559 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:37.556+0000 7efd60937640 -1 reset not still connected to 0x559aee775ee0 2024-04-24T00:59:37.560 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:37.556+0000 7efd60937640 -1 reset not still connected to 0x559af00b4ea0 2024-04-24T00:59:37.563 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:37.560+0000 7efd60937640 -1 reset not still connected to 0x559afbbd3520 2024-04-24T00:59:37.563 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:37.560+0000 7efd60937640 -1 reset not still connected to 0x559b1beb8270 2024-04-24T00:59:37.563 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:37.560+0000 7efd60937640 -1 reset not still connected to 0x559b1c5560d0 2024-04-24T00:59:37.563 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:37.560+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T00:59:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 127342592 bytes 2024-04-24T00:59:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 155097088 bytes 2024-04-24T00:59:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:59:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:59:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_17 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 journal 128M 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_17 128M 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_17 2024-04-24T00:59:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:59:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:59:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:59:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:59:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:59:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_17 2024-04-24T00:59:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T00:59:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:59:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T00:59:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:59:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:59:43.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:59:43.100+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573197bc0c0 finish: resending after timeout 2024-04-24T00:59:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 25019392 bytes 2024-04-24T00:59:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 114491392 bytes 2024-04-24T00:59:48.490 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T00:59:48.491+0000 7efd60937640 -1 reset not still connected to 0x559b0ef8b930 2024-04-24T00:59:48.803 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:59:48.807+0000 7fd951b0e640 -1 reset not still connected to 0x5637198352b0 2024-04-24T00:59:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 170077184 bytes 2024-04-24T00:59:51.748 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:59:51.751+0000 7fd951b0e640 -1 reset not still connected to 0x5637198352b0 2024-04-24T00:59:51.748 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T00:59:51.751+0000 7fd951b0e640 -1 reset not still connected to 0x56371ee280d0 2024-04-24T00:59:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 197351424 bytes 2024-04-24T00:59:52.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:59:52.772+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573197bc0c0 finish: resending after timeout 2024-04-24T00:59:52.822 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T00:59:52.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T00:59:52.823 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-24T00:59:52.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T00:59:52.824+0000 7f0ff08e3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 71782) UID: 1000 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_18 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 journal 128M 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_18 128M 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T00:59:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T00:59:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T00:59:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T00:59:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_18 2024-04-24T00:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T00:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T00:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T00:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T00:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T00:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_18 2024-04-24T00:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T00:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T00:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T00:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T00:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T00:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T00:59:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-04-24T00:59:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 30717952 bytes 2024-04-24T00:59:57.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T00:59:57.772+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573197bc0c0 finish: resending after timeout 2024-04-24T00:59:57.823 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T00:59:57.823 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:00:00.016 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:00.019+0000 7fbc3ce8e640 -1 reset not still connected to 0x556da9e32820 2024-04-24T01:00:00.016 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:00.019+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaedc270 2024-04-24T01:00:00.016 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:00.019+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaeddc70 2024-04-24T01:00:00.017 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:00.019+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dafbeab60 2024-04-24T01:00:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:00:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:00:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_19 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 journal 128M 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_19 128M 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_19 2024-04-24T01:00:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:00:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:00:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:00:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:00:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:00:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_19 2024-04-24T01:00:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:00:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:00:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T01:00:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:00:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:00:03.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:03.164+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573197bc0c0 finish: resending after timeout 2024-04-24T01:00:03.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:03.232+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bb440 finish: resending after timeout 2024-04-24T01:00:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 68443136 bytes 2024-04-24T01:00:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 77984768 bytes 2024-04-24T01:00:07.358 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:07.359+0000 7efd60937640 -1 reset not still connected to 0x559ae322cf70 2024-04-24T01:00:07.360 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:07.363+0000 7efd60937640 -1 reset not still connected to 0x559afb40eea0 2024-04-24T01:00:07.365 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:07.367+0000 7efd60937640 -1 reset not still connected to 0x559afb40f040 2024-04-24T01:00:07.365 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:07.367+0000 7efd60937640 -1 reset not still connected to 0x559afb40f1e0 2024-04-24T01:00:07.365 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:07.367+0000 7efd60937640 -1 reset not still connected to 0x559b03182dd0 2024-04-24T01:00:07.365 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:07.367+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T01:00:07.365 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:07.367+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:00:08.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:08.163+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573197bc0c0 finish: resending after timeout 2024-04-24T01:00:08.776 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:08.775+0000 7efd60937640 -1 reset not still connected to 0x559ae322cf70 2024-04-24T01:00:08.776 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:08.775+0000 7efd60937640 -1 reset not still connected to 0x559afb40eea0 2024-04-24T01:00:08.776 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:08.779+0000 7efd60937640 -1 reset not still connected to 0x559afb40f040 2024-04-24T01:00:08.777 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:08.779+0000 7efd60937640 -1 reset not still connected to 0x559afb40f1e0 2024-04-24T01:00:08.778 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:08.779+0000 7efd60937640 -1 reset not still connected to 0x559b03182dd0 2024-04-24T01:00:08.778 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:08.779+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T01:00:08.778 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:08.779+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:00:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 108491776 bytes 2024-04-24T01:00:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 113845248 bytes 2024-04-24T01:00:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:00:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:00:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:00:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_20 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 journal 128M 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_20 128M 2024-04-24T01:00:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_20 2024-04-24T01:00:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:00:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:00:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:00:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:00:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:00:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_20 2024-04-24T01:00:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:00:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:00:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T01:00:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:00:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:00:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 594944 bytes 2024-04-24T01:00:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 22221824 bytes 2024-04-24T01:00:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 34093056 bytes 2024-04-24T01:00:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 77886464 bytes 2024-04-24T01:00:19.084 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:00:19.083+0000 7ff573346640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 64 2024-04-24T01:00:19.084 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:00:19.083+0000 7ff578b51640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 64 2024-04-24T01:00:19.090 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:00:19.091+0000 7ff57634c640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 60 2024-04-24T01:00:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 106131456 bytes 2024-04-24T01:00:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 123507712 bytes 2024-04-24T01:00:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 137549824 bytes 2024-04-24T01:00:23.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:00:23.091+0000 7f0a9747f640 -1 rbd::mirror::image_sync::SyncPointPruneRequest: 0x557e025ea7e0 send: failed to locate sync point snapshot: .rbd-mirror.4243e5f6-7584-4528-9e21-b1d37acf770f.3aed2e58-4f88-4834-a3a0-59d4bc686232 2024-04-24T01:00:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:00:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_21 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 journal 128M 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_21 128M 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_21 2024-04-24T01:00:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_21 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:00:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:00:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T01:00:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:00:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:00:24.205 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:00:24.206+0000 7fd951b0e640 -1 reset not still connected to 0x563700359930 2024-04-24T01:00:24.206 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:00:24.206+0000 7fd951b0e640 -1 reset not still connected to 0x563703901450 2024-04-24T01:00:24.206 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:00:24.206+0000 7fd951b0e640 -1 reset not still connected to 0x56371a3b1e10 2024-04-24T01:00:24.206 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:00:24.206+0000 7fd951b0e640 -1 reset not still connected to 0x563724a401a0 2024-04-24T01:00:25.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:25.703+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bb440 finish: resending after timeout 2024-04-24T01:00:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 35111936 bytes 2024-04-24T01:00:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:00:26.779+0000 7f0fe98d5640 -1 rbd::mirror::MirrorStatusUpdater 0x56228c112fc0 handle_update_task: failed to update mirror image statuses: (108) Cannot send after transport endpoint shutdown 2024-04-24T01:00:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:00:26.791+0000 7f0fe98d5640 -1 rbd::mirror::InstanceWatcher: 0x56228b52a000 handle_remove_instance_object: error removing rbd_mirror_instance.4456 object: (108) Cannot send after transport endpoint shutdown 2024-04-24T01:00:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:00:26.791+0000 7f0fe90d4640 -1 rbd::mirror::InstanceWatcher: 0x56228b52a000 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2024-04-24T01:00:27.938 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3472654453449363) > weight (0.25) 2024-04-24T01:00:27.938 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7438070491053423) > weight (0.25) 2024-04-24T01:00:27.938 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9240045919472466) > weight (0.25) 2024-04-24T01:00:27.938 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-24T01:00:27.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-24T01:00:27.939 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-24T01:00:27.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:00:27.939+0000 7f2f26897640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 46539) UID: 1000 2024-04-24T01:00:28.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:28.091+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114ba0c0 finish: resending after timeout 2024-04-24T01:00:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 56814592 bytes 2024-04-24T01:00:28.331 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T01:00:28.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T01:00:28.331 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T01:00:28.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T01:00:28.334 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-04-24T01:00:30.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:30.703+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bb440 finish: resending after timeout 2024-04-24T01:00:30.940 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-24T01:00:30.940 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:00:30.941 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-24T01:00:30.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-24T01:00:30.941 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-24T01:00:30.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-24T01:00:30.944 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-04-24T01:00:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 116969472 bytes 2024-04-24T01:00:32.098 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:32.098+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc31aba00 2024-04-24T01:00:32.098 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:32.098+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca2f4340 2024-04-24T01:00:32.098 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:32.098+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T01:00:32.098 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:32.098+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:00:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 135788544 bytes 2024-04-24T01:00:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 143076352 bytes 2024-04-24T01:00:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 146213888 bytes 2024-04-24T01:00:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 190969856 bytes 2024-04-24T01:00:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_22 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 journal 128M 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_22 128M 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_22 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_22 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:00:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:00:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T01:00:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:00:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:00:39.243 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:39.242+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T01:00:39.243 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:39.242+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:00:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 26616832 bytes 2024-04-24T01:00:39.752 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:39.754+0000 7efd60937640 -1 reset not still connected to 0x559b14750820 2024-04-24T01:00:39.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:39.931+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bb440 finish: resending after timeout 2024-04-24T01:00:40.086 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:40.086+0000 7efd60937640 -1 reset not still connected to 0x559b14750820 2024-04-24T01:00:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 62663680 bytes 2024-04-24T01:00:42.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:42.271+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:00:43.563 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:00:43.563+0000 7fbdbe182640 -1 reset not still connected to 0x5622d0269ad0 2024-04-24T01:00:43.563 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:00:43.563+0000 7fbdbe182640 -1 reset not still connected to 0x5622e1a30270 2024-04-24T01:00:43.567 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:00:43.567+0000 7fbdbe182640 -1 reset not still connected to 0x5622ef37a340 2024-04-24T01:00:44.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:44.723+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55731acdda40 finish: resending after timeout 2024-04-24T01:00:44.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:44.931+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bb440 finish: resending after timeout 2024-04-24T01:00:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 137567232 bytes 2024-04-24T01:00:47.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:47.271+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:00:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:00:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_23 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 journal 128M 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_23 128M 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_23 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_23 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:00:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:00:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T01:00:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:00:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:00:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 623616 bytes 2024-04-24T01:00:47.998 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:47.998+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca2f4340 2024-04-24T01:00:48.004 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:48.006+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T01:00:48.004 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:48.006+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:00:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 13603840 bytes 2024-04-24T01:00:49.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:49.931+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bb440 finish: resending after timeout 2024-04-24T01:00:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 24411136 bytes 2024-04-24T01:00:51.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 38298624 bytes 2024-04-24T01:00:51.952 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:51.954+0000 7fbc3ce8e640 -1 reset not still connected to 0x556da9c78d00 2024-04-24T01:00:51.952 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:51.954+0000 7fbc3ce8e640 -1 reset not still connected to 0x556da9e32820 2024-04-24T01:00:51.953 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:51.954+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaedc270 2024-04-24T01:00:51.953 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:51.954+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaeddc70 2024-04-24T01:00:51.953 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:51.954+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dafbeab60 2024-04-24T01:00:51.953 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:51.954+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca97c340 2024-04-24T01:00:52.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:52.271+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:00:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_24 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 journal 128M 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_24 128M 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:00:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_24 2024-04-24T01:00:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:00:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:00:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:00:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:00:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:00:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_24 2024-04-24T01:00:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:00:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:00:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:00:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:00:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T01:00:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:00:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:00:53.213 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:00:53.214+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:00:57.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:00:57.270+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:00:57.816 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:57.813+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:00:57.816 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:57.813+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:00:58.097 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:00:58.097+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 74923008 bytes 2024-04-24T01:01:02.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:02.270+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:01:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_25 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 journal 128M 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:01:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_25 128M 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:01:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_25 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_25 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:01:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:01:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T01:01:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:01:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:01:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 14811136 bytes 2024-04-24T01:01:07.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:07.274+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:01:08.392 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:08.393+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T01:01:08.392 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:08.393+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:01:08.393 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:08.393+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:01:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 55252992 bytes 2024-04-24T01:01:09.762 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:09.761+0000 7efd60937640 -1 reset not still connected to 0x559aee775ee0 2024-04-24T01:01:09.763 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:09.761+0000 7efd60937640 -1 reset not still connected to 0x559af00b4ea0 2024-04-24T01:01:09.763 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:09.761+0000 7efd60937640 -1 reset not still connected to 0x559afbbd3520 2024-04-24T01:01:09.763 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:09.761+0000 7efd60937640 -1 reset not still connected to 0x559b1beb8270 2024-04-24T01:01:09.763 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:09.761+0000 7efd60937640 -1 reset not still connected to 0x559b1c5560d0 2024-04-24T01:01:09.763 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:09.761+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T01:01:10.088 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:10.089+0000 7efd60937640 -1 reset not still connected to 0x559aebbcef70 2024-04-24T01:01:10.088 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:10.089+0000 7efd60937640 -1 reset not still connected to 0x559b02e7fad0 2024-04-24T01:01:10.088 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:10.089+0000 7efd60937640 -1 reset not still connected to 0x559b0ef8b930 2024-04-24T01:01:10.088 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:10.089+0000 7efd60937640 -1 reset not still connected to 0x559b14750820 2024-04-24T01:01:11.761 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:11.761+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb38680 2024-04-24T01:01:11.762 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:11.761+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb736c0 2024-04-24T01:01:11.762 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:11.761+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc31aba00 2024-04-24T01:01:11.762 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:11.761+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca2f4340 2024-04-24T01:01:11.762 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:11.761+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T01:01:11.762 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:11.761+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:01:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 99275776 bytes 2024-04-24T01:01:12.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:12.278+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:01:13.348 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:13.349+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:01:13.350 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:13.349+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97a00 2024-04-24T01:01:13.350 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:13.349+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:01:13.350 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:13.349+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:01:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:01:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:01:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:01:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:01:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_26 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 journal 128M 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_26 128M 2024-04-24T01:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_26 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_26 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:01:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T01:01:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:01:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:01:17.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:17.278+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:01:18.860 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:18.857+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T01:01:18.860 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:18.857+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:01:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_27 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 journal 128M 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_27 128M 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:01:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_27 2024-04-24T01:01:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:01:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:01:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:01:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:01:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:01:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_27 2024-04-24T01:01:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:01:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:01:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T01:01:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:01:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:01:20.827 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:20.825+0000 7fd951b0e640 -1 reset not still connected to 0x56371a3b1e10 2024-04-24T01:01:20.827 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:20.825+0000 7fd951b0e640 -1 reset not still connected to 0x563724a401a0 2024-04-24T01:01:21.798 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:21.797+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:01:21.803 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:21.801+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:01:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 34965504 bytes 2024-04-24T01:01:22.010 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:22.009+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:01:22.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:22.166+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:01:22.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:22.282+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:01:22.640 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:22.637+0000 7efd60937640 -1 reset not still connected to 0x559aee775ee0 2024-04-24T01:01:22.640 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:22.637+0000 7efd60937640 -1 reset not still connected to 0x559afbbd3520 2024-04-24T01:01:22.641 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:22.641+0000 7efd60937640 -1 reset not still connected to 0x559b1beb8270 2024-04-24T01:01:22.641 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:22.641+0000 7efd60937640 -1 reset not still connected to 0x559b1c5560d0 2024-04-24T01:01:22.641 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:22.641+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T01:01:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 60427264 bytes 2024-04-24T01:01:23.299 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:23.297+0000 7efd60937640 -1 reset not still connected to 0x559ae2a5bee0 2024-04-24T01:01:23.304 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:23.305+0000 7efd60937640 -1 reset not still connected to 0x559b0311a680 2024-04-24T01:01:23.304 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:01:23.305+0000 7efd60937640 -1 reset not still connected to 0x559b145cbba0 2024-04-24T01:01:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_28 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 journal 128M 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_28 128M 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:01:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_28 2024-04-24T01:01:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:01:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:01:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:01:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:01:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:01:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_28 2024-04-24T01:01:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:01:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:01:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T01:01:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:01:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:01:27.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:27.281+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:01:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 18404352 bytes 2024-04-24T01:01:28.334 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9948374897815833) > weight (0.25) 2024-04-24T01:01:28.335 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4771545398194442) > weight (0.25) 2024-04-24T01:01:28.335 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.526135460363312) > weight (0.25) 2024-04-24T01:01:28.335 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9973375608480776) > weight (0.25) 2024-04-24T01:01:28.335 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-04-24T01:01:28.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:28.341+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:01:30.072 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:30.069+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db0464340 2024-04-24T01:01:30.072 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:30.069+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:01:30.072 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:30.069+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:01:32.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:32.285+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:01:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 97086464 bytes 2024-04-24T01:01:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 97680384 bytes 2024-04-24T01:01:33.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:33.893+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:01:34.192 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:34.188+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db0464340 2024-04-24T01:01:34.192 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:34.188+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:01:34.199 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:34.196+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97a00 2024-04-24T01:01:34.199 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:34.196+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:01:34.199 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:34.196+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:01:34.912 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:34.908+0000 7fd951b0e640 -1 reset not still connected to 0x56371a1cb110 2024-04-24T01:01:34.912 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:34.908+0000 7fd951b0e640 -1 reset not still connected to 0x56371ee280d0 2024-04-24T01:01:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 130825216 bytes 2024-04-24T01:01:37.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:37.289+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:01:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 196154368 bytes 2024-04-24T01:01:38.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:38.897+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:01:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:01:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:01:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:01:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:01:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_29 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 journal 128M 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_29 128M 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:01:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:01:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:01:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:01:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_29 2024-04-24T01:01:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:01:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:01:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:01:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:01:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:01:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_29 2024-04-24T01:01:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:01:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:01:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T01:01:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:01:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:01:42.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:42.289+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55730fb9ef00 finish: resending after timeout 2024-04-24T01:01:42.544 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:42.540+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:01:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 32249856 bytes 2024-04-24T01:01:43.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:43.901+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:01:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 35936256 bytes 2024-04-24T01:01:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:01:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:01:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_30 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 journal 128M 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_30 128M 2024-04-24T01:01:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_30 2024-04-24T01:01:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:01:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:01:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:01:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:01:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:01:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_30 2024-04-24T01:01:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:01:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:01:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T01:01:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:01:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:01:47.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:47.569+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557310d42300 finish: resending after timeout 2024-04-24T01:01:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 34740224 bytes 2024-04-24T01:01:48.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:48.901+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:01:48.946 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:48.944+0000 7fd951b0e640 -1 reset not still connected to 0x5636fbe1e680 2024-04-24T01:01:48.950 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:48.948+0000 7fd951b0e640 -1 reset not still connected to 0x5636ffb58b60 2024-04-24T01:01:48.950 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:48.948+0000 7fd951b0e640 -1 reset not still connected to 0x56370a887a00 2024-04-24T01:01:48.950 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:48.948+0000 7fd951b0e640 -1 reset not still connected to 0x5637183065b0 2024-04-24T01:01:48.950 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:48.948+0000 7fd951b0e640 -1 reset not still connected to 0x563718d7f2b0 2024-04-24T01:01:48.950 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:48.948+0000 7fd951b0e640 -1 reset not still connected to 0x563719d60410 2024-04-24T01:01:48.951 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:48.948+0000 7fd951b0e640 -1 reset not still connected to 0x563719d60d00 2024-04-24T01:01:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_31 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 journal 128M 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_31 128M 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:01:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:01:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_31 2024-04-24T01:01:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:01:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:01:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:01:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:01:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:01:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_31 2024-04-24T01:01:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:01:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:01:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:01:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:01:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T01:01:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:01:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:01:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 617472 bytes 2024-04-24T01:01:51.012 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:51.012+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:01:51.559 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:51.552+0000 7fd951b0e640 -1 reset not still connected to 0x563719d60410 2024-04-24T01:01:51.559 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:51.552+0000 7fd951b0e640 -1 reset not still connected to 0x563719d60d00 2024-04-24T01:01:51.581 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:51.576+0000 7fd951b0e640 -1 reset not still connected to 0x5636fbe1e680 2024-04-24T01:01:51.581 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:51.576+0000 7fd951b0e640 -1 reset not still connected to 0x56370a887a00 2024-04-24T01:01:51.581 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:01:51.576+0000 7fd951b0e640 -1 reset not still connected to 0x563719d60d00 2024-04-24T01:01:51.599 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:01:51.596+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:01:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 14692352 bytes 2024-04-24T01:01:52.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:52.573+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557310d42300 finish: resending after timeout 2024-04-24T01:01:53.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:53.269+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bbb00 finish: resending after timeout 2024-04-24T01:01:53.558 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:01:53.561+0000 7f63163ce640 -1 reset not still connected to 0x55c557b77e10 2024-04-24T01:01:53.558 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:01:53.561+0000 7f63163ce640 -1 reset not still connected to 0x55c55e877ee0 2024-04-24T01:01:53.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:53.901+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:01:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 47738880 bytes 2024-04-24T01:01:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 84484096 bytes 2024-04-24T01:01:57.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:57.573+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557310d42300 finish: resending after timeout 2024-04-24T01:01:58.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:58.120+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55731acdc6c0 finish: resending after timeout 2024-04-24T01:01:58.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:58.272+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bbb00 finish: resending after timeout 2024-04-24T01:01:58.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:01:58.904+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:01:59.945 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T01:01:59.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T01:01:59.946 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-24T01:01:59.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:01:59.948+0000 7f7264935640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 97258) UID: 1000 2024-04-24T01:02:01.329 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:01.324+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db032fba0 2024-04-24T01:02:01.335 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:01.332+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db5d3e410 2024-04-24T01:02:01.340 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:01.336+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb38680 2024-04-24T01:02:01.340 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:01.336+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb736c0 2024-04-24T01:02:01.340 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:01.336+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc31aba00 2024-04-24T01:02:01.340 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:01.336+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca2f4340 2024-04-24T01:02:01.340 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:01.336+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T01:02:01.340 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:01.336+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:02:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:02:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:02:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:02:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:02:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:02:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_32 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 journal 128M 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_32 128M 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_32 2024-04-24T01:02:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:02:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:02:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:02:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:02:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:02:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_32 2024-04-24T01:02:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:02:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:02:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T01:02:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:02:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:02:01.947 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T01:02:01.947 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:02:01.947 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T01:02:01.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T01:02:01.947 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T01:02:01.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T01:02:01.950 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-04-24T01:02:02.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:02.576+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557310d42300 finish: resending after timeout 2024-04-24T01:02:03.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:03.272+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bbb00 finish: resending after timeout 2024-04-24T01:02:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 27409408 bytes 2024-04-24T01:02:03.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:03.904+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:02:04.900 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:04.896+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db032fba0 2024-04-24T01:02:04.908 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:04.904+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb38680 2024-04-24T01:02:04.908 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:04.904+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb736c0 2024-04-24T01:02:04.908 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:04.904+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc31aba00 2024-04-24T01:02:04.908 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:04.904+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca2f4340 2024-04-24T01:02:04.908 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:04.904+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T01:02:04.908 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:04.904+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:02:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 44973056 bytes 2024-04-24T01:02:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 59842560 bytes 2024-04-24T01:02:07.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:07.576+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557310d42300 finish: resending after timeout 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_33 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 journal 128M 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:02:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_33 128M 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:02:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_33 2024-04-24T01:02:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_33 2024-04-24T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:02:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:02:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T01:02:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:02:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:02:07.998 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:02:08.000+0000 7fbdacd14640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-04-24T01:02:08.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:08.272+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bbb00 finish: resending after timeout 2024-04-24T01:02:08.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:08.904+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:02:12.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:12.576+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557310d42300 finish: resending after timeout 2024-04-24T01:02:13.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:13.276+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bbb00 finish: resending after timeout 2024-04-24T01:02:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 72221696 bytes 2024-04-24T01:02:13.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:13.908+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:02:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 94096384 bytes 2024-04-24T01:02:15.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:15.052+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557319af4b40 finish: resending after timeout 2024-04-24T01:02:17.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:17.576+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557310d42300 finish: resending after timeout 2024-04-24T01:02:18.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:18.276+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bbb00 finish: resending after timeout 2024-04-24T01:02:18.542 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:18.543+0000 7efd60937640 -1 reset not still connected to 0x559afbb896c0 2024-04-24T01:02:18.542 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:18.543+0000 7efd60937640 -1 reset not still connected to 0x559b0311a680 2024-04-24T01:02:18.542 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:18.543+0000 7efd60937640 -1 reset not still connected to 0x559b0f194820 2024-04-24T01:02:18.542 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:18.543+0000 7efd60937640 -1 reset not still connected to 0x559b145cbba0 2024-04-24T01:02:18.561 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:02:18.560+0000 7f63163ce640 -1 reset not still connected to 0x55c53fc5bee0 2024-04-24T01:02:18.561 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:02:18.560+0000 7f63163ce640 -1 reset not still connected to 0x55c55e877ee0 2024-04-24T01:02:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 128177152 bytes 2024-04-24T01:02:18.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:18.908+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:02:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 156780544 bytes 2024-04-24T01:02:20.727 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:02:20.724+0000 7fbdbe182640 -1 reset not still connected to 0x5622d0269ad0 2024-04-24T01:02:20.728 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:02:20.724+0000 7fbdbe182640 -1 reset not still connected to 0x5622e1a30270 2024-04-24T01:02:20.728 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:02:20.724+0000 7fbdbe182640 -1 reset not still connected to 0x5622f5722f70 2024-04-24T01:02:20.751 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:02:20.752+0000 7f63163ce640 -1 reset not still connected to 0x55c532e252b0 2024-04-24T01:02:20.751 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:02:20.752+0000 7f63163ce640 -1 reset not still connected to 0x55c538c94270 2024-04-24T01:02:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:02:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:02:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:02:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_34 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 journal 128M 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_34 128M 2024-04-24T01:02:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_34 2024-04-24T01:02:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:02:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:02:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:02:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:02:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:02:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_34 2024-04-24T01:02:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:02:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:02:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T01:02:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:02:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:02:22.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:22.580+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557310d42300 finish: resending after timeout 2024-04-24T01:02:23.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:23.276+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bbb00 finish: resending after timeout 2024-04-24T01:02:23.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:23.908+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:02:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:02:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:02:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:02:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:02:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_35 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 journal 128M 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_35 128M 2024-04-24T01:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_35 2024-04-24T01:02:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_35 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:02:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:02:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T01:02:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:02:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:02:27.348 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:27.347+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:02:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:27.580+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557310d42300 finish: resending after timeout 2024-04-24T01:02:28.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:28.908+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573198c3bc0 finish: resending after timeout 2024-04-24T01:02:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 36481024 bytes 2024-04-24T01:02:29.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:02:29.419+0000 7fcb0a563640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114ba540 finish: resending after timeout 2024-04-24T01:02:30.078 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:02:30.079+0000 7f630275b640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2024-04-24T01:02:30.090 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:02:30.087+0000 7f6303f5e640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2024-04-24T01:02:30.383 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:30.379+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa099a00 2024-04-24T01:02:30.383 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:30.379+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa7900d0 2024-04-24T01:02:30.383 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:30.379+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa791520 2024-04-24T01:02:30.384 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:30.379+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dacdb4d00 2024-04-24T01:02:30.384 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:30.379+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db2616000 2024-04-24T01:02:30.384 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:30.379+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db5d58a90 2024-04-24T01:02:30.384 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:30.379+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbc15ef70 2024-04-24T01:02:30.384 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:30.379+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc996add0 2024-04-24T01:02:30.384 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:30.379+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dd1a212b0 2024-04-24T01:02:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 69647360 bytes 2024-04-24T01:02:33.465 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:33.463+0000 7efd60937640 -1 reset not still connected to 0x559b1f5b4d00 2024-04-24T01:02:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 91922432 bytes 2024-04-24T01:02:34.938 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:34.939+0000 7efd60937640 -1 reset not still connected to 0x559b0311a680 2024-04-24T01:02:34.938 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:34.939+0000 7efd60937640 -1 reset not still connected to 0x559b0f194820 2024-04-24T01:02:34.938 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:34.939+0000 7efd60937640 -1 reset not still connected to 0x559b145cbba0 2024-04-24T01:02:36.171 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:02:36.171+0000 7fd951b0e640 -1 reset not still connected to 0x563725367380 2024-04-24T01:02:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 135271424 bytes 2024-04-24T01:02:36.681 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:02:36.683+0000 7fd951b0e640 -1 reset not still connected to 0x5637198352b0 2024-04-24T01:02:36.683 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:02:36.683+0000 7fd951b0e640 -1 reset not still connected to 0x56371a1cb110 2024-04-24T01:02:36.685 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:02:36.687+0000 7fd951b0e640 -1 reset not still connected to 0x56371ee280d0 2024-04-24T01:02:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:02:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_36 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 journal 128M 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:02:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_36 128M 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_36 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_36 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:02:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:02:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=8 2024-04-24T01:02:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:02:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:02:40.336 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.673535483338136) > weight (0.25) 2024-04-24T01:02:40.336 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8952346809433591) > weight (0.25) 2024-04-24T01:02:40.336 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6542024290278461) > weight (0.25) 2024-04-24T01:02:40.336 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8572639453015154) > weight (0.25) 2024-04-24T01:02:40.336 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-04-24T01:02:40.607 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:40.607+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db0464340 2024-04-24T01:02:40.607 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:40.607+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:02:40.610 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:40.611+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:02:40.610 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:40.611+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:02:40.610 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:02:40.611+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dd4179d40 2024-04-24T01:02:41.898 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:02:41.899+0000 7f63163ce640 -1 reset not still connected to 0x55c53fc5bee0 2024-04-24T01:02:41.899 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:02:41.899+0000 7f63163ce640 -1 reset not still connected to 0x55c5560c8b60 2024-04-24T01:02:41.899 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:02:41.899+0000 7f63163ce640 -1 reset not still connected to 0x55c55e877ee0 2024-04-24T01:02:42.637 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.634+0000 7efd60937640 -1 reset not still connected to 0x559aeab664e0 2024-04-24T01:02:42.637 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.634+0000 7efd60937640 -1 reset not still connected to 0x559aeb2892b0 2024-04-24T01:02:42.637 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.634+0000 7efd60937640 -1 reset not still connected to 0x559aee775ee0 2024-04-24T01:02:42.637 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.634+0000 7efd60937640 -1 reset not still connected to 0x559af00b4ea0 2024-04-24T01:02:42.637 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.634+0000 7efd60937640 -1 reset not still connected to 0x559afa592a90 2024-04-24T01:02:42.637 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.634+0000 7efd60937640 -1 reset not still connected to 0x559afbbd3520 2024-04-24T01:02:42.644 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.642+0000 7efd60937640 -1 reset not still connected to 0x559b1beb8270 2024-04-24T01:02:42.644 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.642+0000 7efd60937640 -1 reset not still connected to 0x559b1c5560d0 2024-04-24T01:02:42.644 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.642+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T01:02:42.645 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.642+0000 7efd60937640 -1 reset not still connected to 0x559aeab664e0 2024-04-24T01:02:42.645 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.642+0000 7efd60937640 -1 reset not still connected to 0x559aeb2892b0 2024-04-24T01:02:42.645 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.642+0000 7efd60937640 -1 reset not still connected to 0x559aee775ee0 2024-04-24T01:02:42.645 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.642+0000 7efd60937640 -1 reset not still connected to 0x559af00b4ea0 2024-04-24T01:02:42.645 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.642+0000 7efd60937640 -1 reset not still connected to 0x559afa592a90 2024-04-24T01:02:42.645 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.642+0000 7efd60937640 -1 reset not still connected to 0x559afbbd3520 2024-04-24T01:02:42.646 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.646+0000 7efd60937640 -1 reset not still connected to 0x559b1c5560d0 2024-04-24T01:02:42.646 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:42.646+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T01:02:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 41728000 bytes 2024-04-24T01:02:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 43016192 bytes 2024-04-24T01:02:44.792 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:44.790+0000 7efd60937640 -1 reset not still connected to 0x559b03182dd0 2024-04-24T01:02:44.792 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:44.790+0000 7efd60937640 -1 reset not still connected to 0x559b03988340 2024-04-24T01:02:44.792 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:44.790+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T01:02:44.792 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:44.790+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:02:44.792 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:44.790+0000 7efd60937640 -1 reset not still connected to 0x559b145712b0 2024-04-24T01:02:44.792 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:44.790+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:02:44.792 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:44.790+0000 7efd60937640 -1 reset not still connected to 0x559b1f5b4d00 2024-04-24T01:02:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_37 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 journal 128M 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:02:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_37 128M 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_37 2024-04-24T01:02:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_37 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:02:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:02:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T01:02:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:02:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:02:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 48247808 bytes 2024-04-24T01:02:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 103475200 bytes 2024-04-24T01:02:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 115501056 bytes 2024-04-24T01:02:54.521 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:54.518+0000 7efd60937640 -1 reset not still connected to 0x559b1f5b4d00 2024-04-24T01:02:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 161410048 bytes 2024-04-24T01:02:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_38 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 journal 128M 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:02:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_38 128M 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:02:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:02:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:02:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_38 2024-04-24T01:02:57.733 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:57.734+0000 7efd60937640 -1 reset not still connected to 0x559b03182dd0 2024-04-24T01:02:57.733 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:57.734+0000 7efd60937640 -1 reset not still connected to 0x559b03988340 2024-04-24T01:02:57.733 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:57.734+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T01:02:57.733 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:57.734+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:02:57.733 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:57.734+0000 7efd60937640 -1 reset not still connected to 0x559b145712b0 2024-04-24T01:02:57.733 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:57.734+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:02:57.733 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:57.734+0000 7efd60937640 -1 reset not still connected to 0x559b1b7ca750 2024-04-24T01:02:57.733 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:57.734+0000 7efd60937640 -1 reset not still connected to 0x559b1f5b4d00 2024-04-24T01:02:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:02:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:02:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:02:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:02:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:02:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_38 2024-04-24T01:02:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:02:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:02:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:02:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:02:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T01:02:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:02:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:02:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 10095616 bytes 2024-04-24T01:02:59.074 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:59.074+0000 7efd60937640 -1 reset not still connected to 0x559afbb896c0 2024-04-24T01:02:59.074 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:59.074+0000 7efd60937640 -1 reset not still connected to 0x559b0311a680 2024-04-24T01:02:59.074 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:59.074+0000 7efd60937640 -1 reset not still connected to 0x559b0f194820 2024-04-24T01:02:59.075 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:59.074+0000 7efd60937640 -1 reset not still connected to 0x559b145cbba0 2024-04-24T01:02:59.633 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:59.634+0000 7efd60937640 -1 reset not still connected to 0x559afbb896c0 2024-04-24T01:02:59.634 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:59.634+0000 7efd60937640 -1 reset not still connected to 0x559b0311a680 2024-04-24T01:02:59.634 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:02:59.634+0000 7efd60937640 -1 reset not still connected to 0x559b145cbba0 2024-04-24T01:03:03.337 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:03.334+0000 7fd951b0e640 -1 reset not still connected to 0x5637198352b0 2024-04-24T01:03:03.340 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:03.334+0000 7fd951b0e640 -1 reset not still connected to 0x56371ee280d0 2024-04-24T01:03:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 79044608 bytes 2024-04-24T01:03:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 102010880 bytes 2024-04-24T01:03:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 112438272 bytes 2024-04-24T01:03:07.803 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:07.802+0000 7efd60937640 -1 reset not still connected to 0x559b02e7fad0 2024-04-24T01:03:07.803 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:07.802+0000 7efd60937640 -1 reset not still connected to 0x559b0ef8b930 2024-04-24T01:03:07.803 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:07.802+0000 7efd60937640 -1 reset not still connected to 0x559b14750820 2024-04-24T01:03:07.803 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:07.802+0000 7efd60937640 -1 reset not still connected to 0x559b1c4e52b0 2024-04-24T01:03:07.849 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:07.846+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db032fba0 2024-04-24T01:03:07.849 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:07.846+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb38680 2024-04-24T01:03:07.849 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:07.846+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dbbb736c0 2024-04-24T01:03:07.849 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:07.846+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc31aba00 2024-04-24T01:03:07.849 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:07.846+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca2f4340 2024-04-24T01:03:07.849 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:07.846+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T01:03:07.849 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:07.846+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:03:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:03:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:03:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_39 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 journal 128M 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:03:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_39 128M 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_39 2024-04-24T01:03:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:03:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:03:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:03:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:03:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:03:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_39 2024-04-24T01:03:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:03:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:03:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T01:03:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:03:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:03:09.972 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.970+0000 7efd60937640 -1 reset not still connected to 0x559ae322cf70 2024-04-24T01:03:09.973 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.970+0000 7efd60937640 -1 reset not still connected to 0x559afb40f040 2024-04-24T01:03:09.974 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.974+0000 7efd60937640 -1 reset not still connected to 0x559afb40f1e0 2024-04-24T01:03:09.980 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.978+0000 7efd60937640 -1 reset not still connected to 0x559b03182dd0 2024-04-24T01:03:09.981 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.978+0000 7efd60937640 -1 reset not still connected to 0x559b03988340 2024-04-24T01:03:09.981 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.978+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T01:03:09.982 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.982+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:03:09.982 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.982+0000 7efd60937640 -1 reset not still connected to 0x559b145712b0 2024-04-24T01:03:09.982 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.982+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:03:09.983 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.982+0000 7efd60937640 -1 reset not still connected to 0x559b1b7ca750 2024-04-24T01:03:09.983 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:09.982+0000 7efd60937640 -1 reset not still connected to 0x559b1f5b4d00 2024-04-24T01:03:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 25856000 bytes 2024-04-24T01:03:11.882 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:11.878+0000 7fd951b0e640 -1 reset not still connected to 0x5636fbe1e680 2024-04-24T01:03:11.882 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:11.878+0000 7fd951b0e640 -1 reset not still connected to 0x5636ffb58b60 2024-04-24T01:03:11.883 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:11.878+0000 7fd951b0e640 -1 reset not still connected to 0x563702f59450 2024-04-24T01:03:11.883 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:11.878+0000 7fd951b0e640 -1 reset not still connected to 0x56370a887a00 2024-04-24T01:03:11.883 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:11.878+0000 7fd951b0e640 -1 reset not still connected to 0x5637183065b0 2024-04-24T01:03:11.883 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:11.878+0000 7fd951b0e640 -1 reset not still connected to 0x563718d7f2b0 2024-04-24T01:03:11.883 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:11.878+0000 7fd951b0e640 -1 reset not still connected to 0x563719d60410 2024-04-24T01:03:11.883 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:11.878+0000 7fd951b0e640 -1 reset not still connected to 0x563719d60d00 2024-04-24T01:03:11.883 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:11.878+0000 7fd951b0e640 -1 reset not still connected to 0x563725367380 2024-04-24T01:03:12.651 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:12.650+0000 7efd60937640 -1 reset not still connected to 0x559afb40f040 2024-04-24T01:03:12.653 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:12.654+0000 7efd60937640 -1 reset not still connected to 0x559afb40f1e0 2024-04-24T01:03:12.658 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:12.658+0000 7efd60937640 -1 reset not still connected to 0x559b03988340 2024-04-24T01:03:12.658 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:12.658+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T01:03:12.658 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:12.658+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:03:12.658 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:12.658+0000 7efd60937640 -1 reset not still connected to 0x559b145712b0 2024-04-24T01:03:12.658 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:12.658+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:03:12.658 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:12.658+0000 7efd60937640 -1 reset not still connected to 0x559b1b7ca750 2024-04-24T01:03:12.658 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:12.658+0000 7efd60937640 -1 reset not still connected to 0x559b1f5b4d00 2024-04-24T01:03:14.209 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:14.206+0000 7efd60937640 -1 reset not still connected to 0x559afbbd3520 2024-04-24T01:03:14.209 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:14.206+0000 7efd60937640 -1 reset not still connected to 0x559b1beb8270 2024-04-24T01:03:14.209 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:14.206+0000 7efd60937640 -1 reset not still connected to 0x559b1c5560d0 2024-04-24T01:03:14.209 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:14.206+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T01:03:15.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:03:15.458+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5573114bb200 finish: resending after timeout 2024-04-24T01:03:15.558 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:15.558+0000 7efd60937640 -1 reset not still connected to 0x559b02e7fad0 2024-04-24T01:03:15.558 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:15.558+0000 7efd60937640 -1 reset not still connected to 0x559b0ef8b930 2024-04-24T01:03:15.558 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:15.558+0000 7efd60937640 -1 reset not still connected to 0x559b14750820 2024-04-24T01:03:15.559 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:15.558+0000 7efd60937640 -1 reset not still connected to 0x559b1c4e52b0 2024-04-24T01:03:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 106209280 bytes 2024-04-24T01:03:17.773 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:03:17.769+0000 7fd951b0e640 -1 reset not still connected to 0x56371ea3c1a0 2024-04-24T01:03:19.197 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:19.193+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dab1ee9c0 2024-04-24T01:03:19.199 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:19.197+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dafbeab60 2024-04-24T01:03:19.202 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:19.201+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca97c340 2024-04-24T01:03:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_40 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 journal 128M 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_40 128M 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_40 2024-04-24T01:03:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:03:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:03:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:03:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:03:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:03:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_40 2024-04-24T01:03:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:03:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:03:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T01:03:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:03:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:03:19.455 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:03:19.454+0000 7f63163ce640 -1 reset not still connected to 0x55c525c77520 2024-04-24T01:03:19.455 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:03:19.454+0000 7f63163ce640 -1 reset not still connected to 0x55c532e252b0 2024-04-24T01:03:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 22440960 bytes 2024-04-24T01:03:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 34136064 bytes 2024-04-24T01:03:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 34735104 bytes 2024-04-24T01:03:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 58343424 bytes 2024-04-24T01:03:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:03:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:03:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:03:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:03:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:03:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_41 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 journal 128M 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_41 128M 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:03:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:03:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:03:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:03:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_41 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_41 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:03:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:03:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T01:03:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:03:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:03:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 624640 bytes 2024-04-24T01:03:25.982 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:25.977+0000 7efd60937640 -1 reset not still connected to 0x559b03988340 2024-04-24T01:03:25.982 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:25.977+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T01:03:25.982 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:25.977+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:03:25.982 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:25.977+0000 7efd60937640 -1 reset not still connected to 0x559b145712b0 2024-04-24T01:03:25.982 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:25.977+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:03:25.982 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:25.977+0000 7efd60937640 -1 reset not still connected to 0x559b1b7ca750 2024-04-24T01:03:25.983 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:25.977+0000 7efd60937640 -1 reset not still connected to 0x559b1f5b4d00 2024-04-24T01:03:26.194 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:26.193+0000 7efd60937640 -1 reset not still connected to 0x559b02e7fad0 2024-04-24T01:03:26.194 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:26.193+0000 7efd60937640 -1 reset not still connected to 0x559b0ef8b930 2024-04-24T01:03:26.194 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:26.193+0000 7efd60937640 -1 reset not still connected to 0x559b14750820 2024-04-24T01:03:26.194 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:26.193+0000 7efd60937640 -1 reset not still connected to 0x559b1c4e52b0 2024-04-24T01:03:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 31680512 bytes 2024-04-24T01:03:29.667 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:29.665+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db0464340 2024-04-24T01:03:29.667 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:29.665+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:03:29.671 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:29.669+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:03:29.674 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:29.673+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:03:29.674 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:29.673+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dd4179d40 2024-04-24T01:03:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 76644352 bytes 2024-04-24T01:03:33.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:03:33.154+0000 7fcb09d62640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x55731b86e780 finish: resending after timeout 2024-04-24T01:03:34.563 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:34.557+0000 7efd60937640 -1 reset not still connected to 0x559aeba5e820 2024-04-24T01:03:34.564 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:34.561+0000 7efd60937640 -1 reset not still connected to 0x559aebbcef70 2024-04-24T01:03:34.564 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:34.561+0000 7efd60937640 -1 reset not still connected to 0x559b02e7fad0 2024-04-24T01:03:34.564 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:34.561+0000 7efd60937640 -1 reset not still connected to 0x559b0ef8b930 2024-04-24T01:03:34.565 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:34.565+0000 7efd60937640 -1 reset not still connected to 0x559b14750820 2024-04-24T01:03:34.565 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:34.565+0000 7efd60937640 -1 reset not still connected to 0x559b1c4e52b0 2024-04-24T01:03:34.565 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:34.565+0000 7efd60937640 -1 reset not still connected to 0x559aeba5e820 2024-04-24T01:03:34.565 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:34.565+0000 7efd60937640 -1 reset not still connected to 0x559b0ef8b930 2024-04-24T01:03:34.565 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:34.565+0000 7efd60937640 -1 reset not still connected to 0x559b14750820 2024-04-24T01:03:35.009 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559ae322cf70 2024-04-24T01:03:35.009 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559afb40eea0 2024-04-24T01:03:35.009 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559afb40f040 2024-04-24T01:03:35.009 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559afb40f1e0 2024-04-24T01:03:35.010 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559b03182dd0 2024-04-24T01:03:35.010 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559b0e7d2c30 2024-04-24T01:03:35.010 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559b0eaf6820 2024-04-24T01:03:35.010 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559b145712b0 2024-04-24T01:03:35.010 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:03:35.010 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559b15443ad0 2024-04-24T01:03:35.010 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559b1b7ca750 2024-04-24T01:03:35.010 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:35.005+0000 7efd60937640 -1 reset not still connected to 0x559b1f5b4d00 2024-04-24T01:03:35.734 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:03:35.734+0000 7fbdacd14640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2024-04-24T01:03:35.741 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:03:35.738+0000 7ff57634c640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 8 2024-04-24T01:03:35.741 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:03:35.738+0000 7ff57534a640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 6 2024-04-24T01:03:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 135774208 bytes 2024-04-24T01:03:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:03:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:03:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_42 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 journal 128M 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:03:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_42 128M 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_42 2024-04-24T01:03:38.507 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:38.505+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca2f4340 2024-04-24T01:03:38.507 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:38.505+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f9040 2024-04-24T01:03:38.507 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:38.505+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:03:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:03:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:03:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:03:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:03:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:03:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_42 2024-04-24T01:03:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:03:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:03:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T01:03:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:03:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:03:40.220 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:40.217+0000 7fbc3ce8e640 -1 reset not still connected to 0x556da9e32820 2024-04-24T01:03:40.221 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:40.217+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaedc270 2024-04-24T01:03:40.221 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:40.217+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaeddc70 2024-04-24T01:03:40.222 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:40.221+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dab1ee9c0 2024-04-24T01:03:40.222 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:40.221+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dafbeab60 2024-04-24T01:03:40.224 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:40.221+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca97c340 2024-04-24T01:03:40.611 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:40.609+0000 7efd60937640 -1 reset not still connected to 0x559b14750820 2024-04-24T01:03:42.132 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:42.129+0000 7efd60937640 -1 reset not still connected to 0x559b145712b0 2024-04-24T01:03:42.132 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:42.129+0000 7efd60937640 -1 reset not still connected to 0x559b14650f70 2024-04-24T01:03:42.132 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:42.129+0000 7efd60937640 -1 reset not still connected to 0x559b15443ad0 2024-04-24T01:03:42.132 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:03:42.129+0000 7efd60937640 -1 reset not still connected to 0x559b1f5b4d00 2024-04-24T01:03:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 66584576 bytes 2024-04-24T01:03:43.083 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:43.081+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daa791520 2024-04-24T01:03:43.085 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:43.081+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db2616000 2024-04-24T01:03:43.085 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:43.081+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc996add0 2024-04-24T01:03:45.950 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.29350714289227753) > weight (0.25) 2024-04-24T01:03:45.951 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3654439469028756) > weight (0.25) 2024-04-24T01:03:45.951 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3278281987128282) > weight (0.25) 2024-04-24T01:03:45.951 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-24T01:03:45.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-24T01:03:45.951 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-24T01:03:45.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:03:45.949+0000 7fb3ec5c3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 98398) UID: 1000 2024-04-24T01:03:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 98861056 bytes 2024-04-24T01:03:46.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6187000682962325) > weight (0.25) 2024-04-24T01:03:46.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5195005509625488) > weight (0.25) 2024-04-24T01:03:46.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5166291846954815) > weight (0.25) 2024-04-24T01:03:46.337 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-24T01:03:46.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-24T01:03:46.338 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T01:03:46.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:03:46.337+0000 7fcb0ad64640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 29700) UID: 1000 2024-04-24T01:03:46.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:03:46.429+0000 7fcb03555640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x55731b828000 handle_process_entry_safe: failed to commit journal event: (85) Interrupted system call should be restarted 2024-04-24T01:03:46.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:03:46.429+0000 7fcb03555640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x55731b828000 handle_replay_complete: replay encountered an error: (85) Interrupted system call should be restarted 2024-04-24T01:03:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 113924096 bytes 2024-04-24T01:03:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:03:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:03:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:03:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:03:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:03:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_43 2024-04-24T01:03:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 journal 128M 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_43 128M 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:03:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:03:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:03:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:03:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_43 2024-04-24T01:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_43 2024-04-24T01:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:03:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T01:03:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:03:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:03:48.622 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:48.617+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc31aba00 2024-04-24T01:03:48.622 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:48.621+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca2f4340 2024-04-24T01:03:48.622 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:03:48.621+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:03:52.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:03:52.849+0000 7f727785d640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x557b30b10e00 handle_process_entry_safe: failed to commit journal event: (85) Interrupted system call should be restarted 2024-04-24T01:03:52.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:03:52.849+0000 7f727785d640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x557b30b10e00 handle_replay_complete: replay encountered an error: (85) Interrupted system call should be restarted 2024-04-24T01:03:52.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:03:52.853+0000 7f0a93477640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x557e03dfa380 handle_process_entry_safe: failed to commit journal event: (85) Interrupted system call should be restarted 2024-04-24T01:03:52.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:03:52.853+0000 7f0a93477640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x557e03dfa380 handle_replay_complete: replay encountered an error: (85) Interrupted system call should be restarted 2024-04-24T01:03:52.952 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-24T01:03:52.952 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:03:52.953 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-24T01:03:52.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-24T01:03:52.953 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-24T01:03:52.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-24T01:03:52.956 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-04-24T01:03:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 64161792 bytes 2024-04-24T01:03:53.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:03:53.157+0000 7f0a8ac66640 -1 rbd::mirror::ImageReplayer: 0x557e02e27b80 [3/ade9577c-c44a-4c5c-9eca-0d4478f2e2a7] start: already running 2024-04-24T01:03:53.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:03:53.157+0000 7f0a8ac66640 -1 rbd::mirror::ImageReplayer: 0x557e03cbea00 [3/fc6edd80-fe19-4761-8511-0b207141c6b4] start: already running 2024-04-24T01:03:53.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:03:53.157+0000 7f0a8ac66640 -1 rbd::mirror::ImageReplayer: 0x557e03cbec80 [3/bf7d5dd9-3a26-484b-b4a3-1b53efd4662a] start: already running 2024-04-24T01:03:53.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:03:53.161+0000 7f727084f640 -1 rbd::mirror::ImageReplayer: 0x557b31b88000 [3/d839cf7d-cf05-4344-a46e-941d5468574b] start: already running 2024-04-24T01:03:53.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:03:53.161+0000 7f727084f640 -1 rbd::mirror::ImageReplayer: 0x557b2fe51b80 [3/c63da6a1-db3e-4796-8aef-3d380b5b78e5] start: already running 2024-04-24T01:03:53.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:03:53.161+0000 7f727084f640 -1 rbd::mirror::ImageReplayer: 0x557b30431b80 [3/fa1ff108-8ada-4ac8-ac25-88a635c25c42] start: already running 2024-04-24T01:03:53.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:03:53.161+0000 7f727084f640 -1 rbd::mirror::ImageReplayer: 0x557b30431400 [3/f6a308e5-3690-4fda-807d-7defdc598adc] start: already running 2024-04-24T01:03:53.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:03:53.189+0000 7f7271050640 -1 rbd::mirror::ImageReplayer: 0x557b31b88780 [3/d9ef14ae-65bb-4ce7-b745-30b1926864dd] start: already running 2024-04-24T01:03:53.577 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:03:53.577+0000 7fbdbe182640 -1 reset not still connected to 0x5622e587fe10 2024-04-24T01:03:53.577 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:03:53.577+0000 7fbdbe182640 -1 reset not still connected to 0x5622f71b40d0 2024-04-24T01:03:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 64784384 bytes 2024-04-24T01:03:53.709 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:03:53.709+0000 7fbdbe182640 -1 reset not still connected to 0x5622e222fe10 2024-04-24T01:03:54.338 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-24T01:03:54.338 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:03:54.339 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-24T01:03:54.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-24T01:03:54.339 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-24T01:03:54.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-24T01:03:54.341 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-04-24T01:03:54.874 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:03:54.877+0000 7f6307765640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-04-24T01:03:54.878 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:03:54.877+0000 7f6306763640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2024-04-24T01:03:54.883 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:03:54.885+0000 7f630375d640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2024-04-24T01:03:54.909 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:03:54.897+0000 7fbdacd14640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-04-24T01:03:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 80060416 bytes 2024-04-24T01:03:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:03:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:03:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:03:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:03:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:03:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_44 2024-04-24T01:03:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 journal 128M 2024-04-24T01:03:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_44 128M 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:03:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:03:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:03:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:03:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:03:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_44 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_44 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:03:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:03:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T01:03:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:03:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:03:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 21155840 bytes 2024-04-24T01:04:00.789 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:00.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc1ff5520 2024-04-24T01:04:00.790 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:00.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc2455d40 2024-04-24T01:04:00.790 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:00.784+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:04:00.791 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:00.788+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97a00 2024-04-24T01:04:00.791 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:00.788+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:04:00.791 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:00.788+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:04:00.791 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:00.788+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de1885a00 2024-04-24T01:04:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 48513024 bytes 2024-04-24T01:04:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:04:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:04:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_45 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 journal 128M 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:04:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_45 128M 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_45 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_45 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:04:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:04:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T01:04:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:04:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:04:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 9486336 bytes 2024-04-24T01:04:05.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:05.793+0000 7f727d869640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832600 finish: resending after timeout 2024-04-24T01:04:05.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:05.793+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832d80 finish: resending after timeout 2024-04-24T01:04:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 99167232 bytes 2024-04-24T01:04:10.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:10.793+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832600 finish: resending after timeout 2024-04-24T01:04:10.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:10.793+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832d80 finish: resending after timeout 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_46 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 journal 128M 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_46 128M 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:04:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_46 2024-04-24T01:04:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:04:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:04:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:04:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:04:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:04:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_46 2024-04-24T01:04:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:04:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:04:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T01:04:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:04:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:04:12.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:12.717+0000 7f727d869640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b31525200 finish: resending after timeout 2024-04-24T01:04:15.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:15.801+0000 7f727d869640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832600 finish: resending after timeout 2024-04-24T01:04:15.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:15.801+0000 7f727d869640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832d80 finish: resending after timeout 2024-04-24T01:04:16.634 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:04:16.632+0000 7f6305f62640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 24 2024-04-24T01:04:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 67649536 bytes 2024-04-24T01:04:17.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:17.720+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b31525200 finish: resending after timeout 2024-04-24T01:04:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 125641728 bytes 2024-04-24T01:04:20.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:20.800+0000 7f727d869640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832d80 finish: resending after timeout 2024-04-24T01:04:20.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:20.800+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832600 finish: resending after timeout 2024-04-24T01:04:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 147054592 bytes 2024-04-24T01:04:22.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:22.720+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b31525200 finish: resending after timeout 2024-04-24T01:04:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:04:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_47 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 journal 128M 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_47 128M 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:04:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:04:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:04:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_47 2024-04-24T01:04:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:04:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:04:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:04:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:04:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:04:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_47 2024-04-24T01:04:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:04:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:04:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T01:04:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:04:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:04:26.112 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:04:26.112+0000 7f630375d640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 10 2024-04-24T01:04:26.757 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:04:26.756+0000 7f6306763640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-04-24T01:04:26.757 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:04:26.756+0000 7f630475f640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2024-04-24T01:04:26.810 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:04:26.812+0000 7fbdaa50f640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-04-24T01:04:26.813 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:04:26.812+0000 7fbdacd14640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 11 2024-04-24T01:04:26.815 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:04:26.816+0000 7fbdad515640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 14 2024-04-24T01:04:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 41930752 bytes 2024-04-24T01:04:27.781 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:04:27.780+0000 7f63163ce640 -1 reset not still connected to 0x55c55503dd40 2024-04-24T01:04:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 54572032 bytes 2024-04-24T01:04:28.881 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:28.880+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:04:28.889 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:28.884+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de1885a00 2024-04-24T01:04:28.890 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:28.884+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de407bba0 2024-04-24T01:04:29.310 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:29.308+0000 7fbc3ce8e640 -1 reset not still connected to 0x556da9e32820 2024-04-24T01:04:29.311 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:29.308+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaeddc70 2024-04-24T01:04:29.318 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:29.308+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dac331ba0 2024-04-24T01:04:29.318 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:29.308+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db2ceaea0 2024-04-24T01:04:29.318 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:29.308+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc80b7790 2024-04-24T01:04:29.318 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:29.308+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de3bb4340 2024-04-24T01:04:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 68294656 bytes 2024-04-24T01:04:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 85539840 bytes 2024-04-24T01:04:31.876 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:04:31.876+0000 7f6307765640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 11 2024-04-24T01:04:31.877 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:04:31.876+0000 7f630475f640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 13 2024-04-24T01:04:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 99922944 bytes 2024-04-24T01:04:33.092 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:33.088+0000 7efd60937640 -1 reset not still connected to 0x559ae8d54f70 2024-04-24T01:04:33.093 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:33.088+0000 7efd60937640 -1 reset not still connected to 0x559af8caeb60 2024-04-24T01:04:33.095 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:33.088+0000 7efd60937640 -1 reset not still connected to 0x559b03cb0750 2024-04-24T01:04:33.095 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:33.088+0000 7efd60937640 -1 reset not still connected to 0x559b145cbba0 2024-04-24T01:04:33.096 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:33.088+0000 7efd60937640 -1 reset not still connected to 0x559b1edca750 2024-04-24T01:04:33.096 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:33.092+0000 7efd60937640 -1 reset not still connected to 0x559b1ef3add0 2024-04-24T01:04:33.096 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:33.092+0000 7efd60937640 -1 reset not still connected to 0x559b2074f520 2024-04-24T01:04:33.096 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:33.092+0000 7efd60937640 -1 reset not still connected to 0x559b20757380 2024-04-24T01:04:33.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:33.888+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832d80 finish: resending after timeout 2024-04-24T01:04:33.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:33.888+0000 7f727d869640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b30832600 finish: resending after timeout 2024-04-24T01:04:33.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:33.888+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b31525200 finish: resending after timeout 2024-04-24T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:04:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_48 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 journal 128M 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:04:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_48 128M 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_48 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_48 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:04:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:04:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T01:04:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:04:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:04:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 19778560 bytes 2024-04-24T01:04:39.002 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:38.999+0000 7efd60937640 -1 reset not still connected to 0x559b1f563ad0 2024-04-24T01:04:39.003 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:39.003+0000 7efd60937640 -1 reset not still connected to 0x559b1f563ba0 2024-04-24T01:04:39.007 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:39.007+0000 7efd60937640 -1 reset not still connected to 0x559b1f563d40 2024-04-24T01:04:39.377 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:39.379+0000 7fd951b0e640 -1 reset not still connected to 0x56370df24270 2024-04-24T01:04:39.382 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:39.383+0000 7fd951b0e640 -1 reset not still connected to 0x56370df24340 2024-04-24T01:04:39.382 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:39.383+0000 7fd951b0e640 -1 reset not still connected to 0x5637273ea410 2024-04-24T01:04:39.382 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:39.383+0000 7fd951b0e640 -1 reset not still connected to 0x5637273eb110 2024-04-24T01:04:39.383 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:39.383+0000 7fd951b0e640 -1 reset not still connected to 0x56372d310410 2024-04-24T01:04:39.383 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:39.383+0000 7fd951b0e640 -1 reset not still connected to 0x56372ff93d40 2024-04-24T01:04:39.383 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:39.383+0000 7fd951b0e640 -1 reset not still connected to 0x5637335d1930 2024-04-24T01:04:40.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:40.164+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b31525200 finish: resending after timeout 2024-04-24T01:04:40.738 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:40.735+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc1ff5520 2024-04-24T01:04:40.738 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:40.735+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc2455d40 2024-04-24T01:04:40.738 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:40.735+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:04:40.738 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:40.735+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:04:40.739 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:40.739+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:04:40.739 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:40.739+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de1885a00 2024-04-24T01:04:40.739 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:40.739+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de407bba0 2024-04-24T01:04:44.272 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:44.271+0000 7efd60937640 -1 reset not still connected to 0x559b1ef3be10 2024-04-24T01:04:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 163480576 bytes 2024-04-24T01:04:45.760 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:45.759+0000 7efd60937640 -1 reset not still connected to 0x559b1edca8f0 2024-04-24T01:04:45.760 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:45.759+0000 7efd60937640 -1 reset not still connected to 0x559b1edcdee0 2024-04-24T01:04:45.760 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:45.759+0000 7efd60937640 -1 reset not still connected to 0x559b1f563ad0 2024-04-24T01:04:45.760 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:45.759+0000 7efd60937640 -1 reset not still connected to 0x559b1f563d40 2024-04-24T01:04:46.329 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:46.327+0000 7efd60937640 -1 reset not still connected to 0x559b07134270 2024-04-24T01:04:46.338 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:46.335+0000 7efd60937640 -1 reset not still connected to 0x559b1edca8f0 2024-04-24T01:04:46.338 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:46.335+0000 7efd60937640 -1 reset not still connected to 0x559b1edcdee0 2024-04-24T01:04:46.338 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:46.335+0000 7efd60937640 -1 reset not still connected to 0x559b1f563ad0 2024-04-24T01:04:46.338 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:46.335+0000 7efd60937640 -1 reset not still connected to 0x559b1f563ba0 2024-04-24T01:04:46.338 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:46.335+0000 7efd60937640 -1 reset not still connected to 0x559b1f563d40 2024-04-24T01:04:47.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:47.308+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b31525200 finish: resending after timeout 2024-04-24T01:04:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_49 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 journal 128M 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_49 128M 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_49 2024-04-24T01:04:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:04:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:04:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:04:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:04:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:04:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_49 2024-04-24T01:04:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:04:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:04:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T01:04:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:04:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:04:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 7581696 bytes 2024-04-24T01:04:49.594 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:49.595+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de3bb4340 2024-04-24T01:04:50.311 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:50.311+0000 7efd60937640 -1 reset not still connected to 0x559af8113040 2024-04-24T01:04:50.314 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:50.315+0000 7efd60937640 -1 reset not still connected to 0x559b1ef37930 2024-04-24T01:04:50.314 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:50.315+0000 7efd60937640 -1 reset not still connected to 0x559b1ef3be10 2024-04-24T01:04:50.684 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:50.683+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T01:04:52.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:04:52.312+0000 7f727d869640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b31525200 finish: resending after timeout 2024-04-24T01:04:53.309 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:53.307+0000 7fd951b0e640 -1 reset not still connected to 0x563709be6680 2024-04-24T01:04:53.311 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:53.311+0000 7fd951b0e640 -1 reset not still connected to 0x56370ed61860 2024-04-24T01:04:53.312 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:53.311+0000 7fd951b0e640 -1 reset not still connected to 0x56370ed61d40 2024-04-24T01:04:53.312 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:53.311+0000 7fd951b0e640 -1 reset not still connected to 0x5637273ea5b0 2024-04-24T01:04:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 72519680 bytes 2024-04-24T01:04:54.440 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:54.435+0000 7fd951b0e640 -1 reset not still connected to 0x56370ed61e10 2024-04-24T01:04:54.440 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:54.439+0000 7fd951b0e640 -1 reset not still connected to 0x563719a5af70 2024-04-24T01:04:54.440 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:54.439+0000 7fd951b0e640 -1 reset not still connected to 0x56371a1cac30 2024-04-24T01:04:55.343 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7576710328455598) > weight (0.25) 2024-04-24T01:04:55.343 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5113195203139485) > weight (0.25) 2024-04-24T01:04:55.343 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5672183559051787) > weight (0.25) 2024-04-24T01:04:55.343 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-24T01:04:55.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-24T01:04:55.343 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T01:04:55.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:04:55.344+0000 7f049b277640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 100966) UID: 1000 2024-04-24T01:04:55.660 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:55.655+0000 7fd951b0e640 -1 reset not still connected to 0x563709be7520 2024-04-24T01:04:55.661 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:55.663+0000 7fd951b0e640 -1 reset not still connected to 0x563709be7930 2024-04-24T01:04:55.661 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:55.663+0000 7fd951b0e640 -1 reset not still connected to 0x56370ed61e10 2024-04-24T01:04:55.662 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:55.663+0000 7fd951b0e640 -1 reset not still connected to 0x563719a5af70 2024-04-24T01:04:55.662 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:04:55.663+0000 7fd951b0e640 -1 reset not still connected to 0x56371a1cac30 2024-04-24T01:04:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 100378624 bytes 2024-04-24T01:04:57.101 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:57.099+0000 7efd60937640 -1 reset not still connected to 0x559b15440820 2024-04-24T01:04:57.101 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:57.099+0000 7efd60937640 -1 reset not still connected to 0x559b1ef37930 2024-04-24T01:04:57.101 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:57.099+0000 7efd60937640 -1 reset not still connected to 0x559b1ef3be10 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_50 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 journal 128M 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n journal ']' 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_50 128M 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:04:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T01:04:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T01:04:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:04:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_50 2024-04-24T01:04:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ is_pool_mirror_mode_image mirror 2024-04-24T01:04:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' pool = image ']' 2024-04-24T01:04:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ case "${pool}" in 2024-04-24T01:04:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T01:04:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T01:04:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_50 2024-04-24T01:04:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:04:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:04:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:04:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:04:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T01:04:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:04:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:04:57.826 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:04:57.823+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de3bb45b0 2024-04-24T01:04:58.116 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:58.115+0000 7efd60937640 -1 reset not still connected to 0x559b15440820 2024-04-24T01:04:58.116 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:58.115+0000 7efd60937640 -1 reset not still connected to 0x559b1ef37930 2024-04-24T01:04:58.116 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:58.115+0000 7efd60937640 -1 reset not still connected to 0x559b1ef3be10 2024-04-24T01:04:58.278 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:58.279+0000 7efd60937640 -1 reset not still connected to 0x559b04dbde10 2024-04-24T01:04:58.278 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:04:58.279+0000 7efd60937640 -1 reset not still connected to 0x559b15803380 2024-04-24T01:05:00.879 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:05:00.879+0000 7ff5877ba640 -1 reset not still connected to 0x5613ef36f380 2024-04-24T01:05:00.879 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:05:00.879+0000 7ff5877ba640 -1 reset not still connected to 0x5613f79c6b60 2024-04-24T01:05:00.880 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:05:00.879+0000 7ff5877ba640 -1 reset not still connected to 0x56140107d1e0 2024-04-24T01:05:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 35809280 bytes 2024-04-24T01:05:02.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:05:02.271+0000 7f727e06a640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b31525200 finish: resending after timeout 2024-04-24T01:05:02.957 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5850376744544231) > weight (0.25) 2024-04-24T01:05:02.957 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2567758406274856) > weight (0.25) 2024-04-24T01:05:02.957 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-24T01:05:02.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-24T01:05:02.958 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-24T01:05:02.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:05:02.959+0000 7f140a188640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 29775) UID: 1000 2024-04-24T01:05:03.344 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-24T01:05:03.345 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:05:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 70611968 bytes 2024-04-24T01:05:03.708 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:03.703+0000 7fd951b0e640 -1 reset not still connected to 0x5637036b75f0 2024-04-24T01:05:03.708 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:03.707+0000 7fd951b0e640 -1 reset not still connected to 0x5637097460d0 2024-04-24T01:05:03.708 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:03.707+0000 7fd951b0e640 -1 reset not still connected to 0x563709be6000 2024-04-24T01:05:03.708 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:03.707+0000 7fd951b0e640 -1 reset not still connected to 0x563709be61a0 2024-04-24T01:05:03.708 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:03.707+0000 7fd951b0e640 -1 reset not still connected to 0x563709be6680 2024-04-24T01:05:03.708 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:03.707+0000 7fd951b0e640 -1 reset not still connected to 0x563709be68f0 2024-04-24T01:05:03.708 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:03.707+0000 7fd951b0e640 -1 reset not still connected to 0x56370ed61860 2024-04-24T01:05:03.708 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:03.707+0000 7fd951b0e640 -1 reset not still connected to 0x56370ed61d40 2024-04-24T01:05:03.708 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:03.707+0000 7fd951b0e640 -1 reset not still connected to 0x5637273ea5b0 2024-04-24T01:05:07.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:05:07.271+0000 7f727d869640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x557b31525200 finish: resending after timeout 2024-04-24T01:05:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_count=50 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local count 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_count=0 2024-04-24T01:05:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:05:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T01:05:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T01:05:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T01:05:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-24T01:05:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'images: ' 2024-04-24T01:05:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 2 2024-04-24T01:05:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T01:05:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ count=50 2024-04-24T01:05:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 50 = 50 2024-04-24T01:05:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-04-24T01:05:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:05:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state 2024-04-24T01:05:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T01:05:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T01:05:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T01:05:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-24T01:05:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'image health:' 2024-04-24T01:05:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 3 2024-04-24T01:05:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T01:05:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=WARNING 2024-04-24T01:05:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test WARNING = ERROR 2024-04-24T01:05:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test WARNING = OK 2024-04-24T01:05:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T01:05:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 -ne 1 2024-04-24T01:05:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 30 2024-04-24T01:05:07.959 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-24T01:05:07.959 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:05:07.959 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-24T01:05:07.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-24T01:05:07.960 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-24T01:05:07.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-24T01:05:07.962 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-04-24T01:05:08.299 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:08.299+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de188fee0 2024-04-24T01:05:08.304 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:08.303+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de3bb4340 2024-04-24T01:05:08.307 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:08.307+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de3bb45b0 2024-04-24T01:05:08.661 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:08.663+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db032fba0 2024-04-24T01:05:09.446 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-24T01:05:09.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-24T01:05:09.446 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-24T01:05:09.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-24T01:05:09.449 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-04-24T01:05:12.021 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:12.023+0000 7efd60937640 -1 reset not still connected to 0x559ae9baa410 2024-04-24T01:05:12.022 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:12.023+0000 7efd60937640 -1 reset not still connected to 0x559aeed8c270 2024-04-24T01:05:12.146 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:05:12.143+0000 7fbdbe182640 -1 reset not still connected to 0x5622fbb031e0 2024-04-24T01:05:12.147 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:05:12.147+0000 7fbdbe182640 -1 reset not still connected to 0x56230ce12c30 2024-04-24T01:05:14.747 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:14.747+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db2ce8f70 2024-04-24T01:05:14.747 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:14.747+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc1ff5520 2024-04-24T01:05:14.747 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:14.747+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc2455d40 2024-04-24T01:05:14.749 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:14.747+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:05:14.749 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:14.747+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97a00 2024-04-24T01:05:14.749 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:14.747+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:05:14.749 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:14.747+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:05:14.749 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:14.747+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de1885a00 2024-04-24T01:05:14.749 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:14.747+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de407bba0 2024-04-24T01:05:14.911 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:14.907+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T01:05:14.911 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:14.907+0000 7efd60937640 -1 reset not still connected to 0x559b1edc5ba0 2024-04-24T01:05:16.271 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:16.271+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc1ff5520 2024-04-24T01:05:16.271 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:16.271+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc2455d40 2024-04-24T01:05:16.272 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:16.271+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:05:16.277 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:16.275+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97a00 2024-04-24T01:05:16.277 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:16.275+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:05:16.278 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:16.275+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:05:16.278 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:16.275+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de1885a00 2024-04-24T01:05:16.278 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:16.275+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de407bba0 2024-04-24T01:05:18.258 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:18.259+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:05:18.258 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:18.259+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de16f8d00 2024-04-24T01:05:18.258 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:18.259+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de43dfba0 2024-04-24T01:05:21.444 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559ae8d54f70 2024-04-24T01:05:21.444 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559aeb72cb60 2024-04-24T01:05:21.444 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559af5a27e10 2024-04-24T01:05:21.445 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559af8caeb60 2024-04-24T01:05:21.445 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559b03cb0750 2024-04-24T01:05:21.445 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559b1449dd40 2024-04-24T01:05:21.445 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559b145cbba0 2024-04-24T01:05:21.445 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559b1edca750 2024-04-24T01:05:21.445 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559b1ef3add0 2024-04-24T01:05:21.445 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559b2074f520 2024-04-24T01:05:21.445 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:21.442+0000 7efd60937640 -1 reset not still connected to 0x559b20757380 2024-04-24T01:05:24.246 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:24.242+0000 7fd951b0e640 -1 reset not still connected to 0x56370df249c0 2024-04-24T01:05:27.793 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:27.794+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db032fba0 2024-04-24T01:05:27.797 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:27.794+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db27ed6c0 2024-04-24T01:05:27.799 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:27.794+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc31aba00 2024-04-24T01:05:27.800 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:27.794+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca9f92b0 2024-04-24T01:05:27.800 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:27.794+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de16f8d00 2024-04-24T01:05:27.800 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:27.794+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de43dfba0 2024-04-24T01:05:28.595 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:28.594+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T01:05:28.601 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:28.594+0000 7efd60937640 -1 reset not still connected to 0x559b1edc5ba0 2024-04-24T01:05:28.602 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:28.602+0000 7efd60937640 -1 reset not still connected to 0x559af15b31e0 2024-04-24T01:05:28.602 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:28.602+0000 7efd60937640 -1 reset not still connected to 0x559b04dbde10 2024-04-24T01:05:28.602 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:28.602+0000 7efd60937640 -1 reset not still connected to 0x559b1c571ba0 2024-04-24T01:05:28.602 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:28.602+0000 7efd60937640 -1 reset not still connected to 0x559b1edc5ba0 2024-04-24T01:05:28.776 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:28.774+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dafbc3ad0 2024-04-24T01:05:28.776 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:28.774+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de16c5450 2024-04-24T01:05:28.776 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:28.774+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de29052b0 2024-04-24T01:05:28.778 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:05:28.778+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de29184e0 2024-04-24T01:05:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-24T01:05:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'image health:' 2024-04-24T01:05:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 3 2024-04-24T01:05:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T01:05:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=OK 2024-04-24T01:05:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = ERROR 2024-04-24T01:05:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = OK 2024-04-24T01:05:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-04-24T01:05:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_count=50 2024-04-24T01:05:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:05:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local count 2024-04-24T01:05:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_count=0 2024-04-24T01:05:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:05:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T01:05:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T01:05:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T01:05:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-24T01:05:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'images: ' 2024-04-24T01:05:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 2 2024-04-24T01:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T01:05:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ count=50 2024-04-24T01:05:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 50 = 50 2024-04-24T01:05:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-04-24T01:05:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:05:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state 2024-04-24T01:05:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T01:05:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T01:05:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T01:05:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-24T01:05:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'image health:' 2024-04-24T01:05:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 3 2024-04-24T01:05:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T01:05:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=OK 2024-04-24T01:05:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = ERROR 2024-04-24T01:05:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = OK 2024-04-24T01:05:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: compare many images' 2024-04-24T01:05:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.aEE7yWfHIo/rbd-mirror.test.log 2024-04-24T01:05:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T01:05:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 01:05:37 TEST: compare many images 2024-04-24T01:05:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 01:05:37 TEST: compare many images 2024-04-24T01:05:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 50 2024-04-24T01:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T01:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_1 snap 2024-04-24T01:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:05:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-04-24T01:05:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-04-24T01:05:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:05:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:05:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:05:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-04-24T01:05:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:05:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:05:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:05:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-04-24T01:05:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:05:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:05:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:05:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:05:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:05:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:05:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-04-24T01:05:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:05:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:05:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:05:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-04-24T01:05:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-24T01:05:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:05:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:05:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:05:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-04-24T01:05:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_11 2024-04-24T01:05:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_13 2024-04-24T01:05:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_19 2024-04-24T01:05:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:05:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:05:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:05:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:05:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-04-24T01:05:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:05:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_15 2024-04-24T01:05:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_17 2024-04-24T01:05:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_1 2024-04-24T01:05:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:05:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:05:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:05:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:05:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:05:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:05:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:05:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:05:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:05:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_1 2024-04-24T01:05:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:05:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:05:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:05:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:05:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:05:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:05:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_1 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:05:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:05:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:05:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:05:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:05:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_1 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_1 ']' 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:05:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:05:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:05:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:05:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-24T01:05:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:05:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-04-24T01:05:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:05:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:05:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_11 2024-04-24T01:05:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_13 2024-04-24T01:05:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_19 2024-04-24T01:05:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_1 2024-04-24T01:05:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:05:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:05:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:05:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:05:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:05:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_15 2024-04-24T01:05:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_17 2024-04-24T01:05:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_1 2024-04-24T01:05:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:05:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:05:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:05:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:05:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:05:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:05:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-04-24T01:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-04-24T01:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-24T01:05:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:05:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-24T01:05:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:05:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:05:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:05:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:05:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:05:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:05:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T01:05:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:05:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:05:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:05:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:05:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:05:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:05:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=1 2024-04-24T01:05:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-24T01:05:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:05:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:05:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:05:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_1 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_1 ']' 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:05:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:05:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-04-24T01:05:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:05:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-24T01:05:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:05:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:05:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:05:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:05:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_11 2024-04-24T01:05:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_13 2024-04-24T01:05:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_19 2024-04-24T01:05:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_1 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:05:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:05:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:05:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_15 2024-04-24T01:05:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_17 2024-04-24T01:05:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_1 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:05:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:05:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:05:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-04-24T01:05:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-04-24T01:05:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:05:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:05:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-04-24T01:05:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-04-24T01:05:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:05:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:05:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:05:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_1 2024-04-24T01:05:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:05:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_1 ']' 2024-04-24T01:05:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-04-24T01:05:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-04-24T01:05:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:05:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:05:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:05:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:05:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:05:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-04-24T01:05:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:05:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:05:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:05:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:05:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-24T01:05:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:05:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-04-24T01:05:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:05:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:05:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:05:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:05:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:05:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_11 2024-04-24T01:05:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_13 2024-04-24T01:05:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_19 2024-04-24T01:05:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_1 2024-04-24T01:05:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:05:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:05:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:05:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_15 2024-04-24T01:05:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_17 2024-04-24T01:05:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_1 2024-04-24T01:05:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:05:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:05:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:05:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-04-24T01:05:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-04-24T01:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:05:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-04-24T01:05:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-04-24T01:05:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:05:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:05:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:05:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_1 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_1 ']' 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:05:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:05:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:05:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:05:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-04-24T01:05:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:05:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:05:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-24T01:05:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:05:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:05:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:05:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_11 2024-04-24T01:05:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_13 2024-04-24T01:05:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_19 2024-04-24T01:05:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_1 2024-04-24T01:05:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:05:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:05:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:05:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_15 2024-04-24T01:05:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_17 2024-04-24T01:05:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_1 2024-04-24T01:05:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:05:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:05:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:05:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-04-24T01:05:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-04-24T01:05:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:05:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:44.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:44.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:05:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-04-24T01:05:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-04-24T01:05:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:05:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:05:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:05:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_1 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_1 ']' 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:05:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:05:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:05:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:05:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-04-24T01:05:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:05:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:05:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:05:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-24T01:05:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:05:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:05:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_11 2024-04-24T01:05:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_13 2024-04-24T01:05:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_19 2024-04-24T01:05:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_1 2024-04-24T01:05:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:05:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:05:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:05:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:05:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:05:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_15 2024-04-24T01:05:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_17 2024-04-24T01:05:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_1 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:05:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:05:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:05:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:05:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:05:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:05:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:05:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:05:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_1 2024-04-24T01:05:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-04-24T01:05:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-04-24T01:05:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:05:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:05:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-24T01:05:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-24T01:05:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:05:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:05:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6075], [object_number=6, tag_tid=1, entry_tid=6074]]], state=connected] 2024-04-24T01:05:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.status 2024-04-24T01:05:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:05:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:05:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-04-24T01:05:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:05:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:05:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:05:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:05:48.316 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:48.314+0000 7efd60937640 -1 reset not still connected to 0x559af8c6e680 2024-04-24T01:05:48.316 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:48.314+0000 7efd60937640 -1 reset not still connected to 0x559af8caea90 2024-04-24T01:05:48.316 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:48.314+0000 7efd60937640 -1 reset not still connected to 0x559b1296a5b0 2024-04-24T01:05:48.316 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:48.314+0000 7efd60937640 -1 reset not still connected to 0x559b1437aa90 2024-04-24T01:05:48.317 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:05:48.314+0000 7efd60937640 -1 reset not still connected to 0x559b144b0270 2024-04-24T01:05:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-04-24T01:05:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_1': 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d85396f7b5 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d85396f7b5 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:57:02 2024 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:57:02 2024 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:58:03 2024 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:05:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d85396f7b5 2024-04-24T01:05:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:05:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:05:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 38a60051-6fd3-41f5-9f10-5deaea36d78a 2024-04-24T01:05:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:05:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_1 snap 2024-04-24T01:05:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:05:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:05:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:05:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.export 2024-04-24T01:05:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_1.export 2024-04-24T01:05:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_1.export 2024-04-24T01:05:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.export 2024-04-24T01:05:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:05:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_1.export 2024-04-24T01:05:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:05:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_1.export 2024-04-24T01:05:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_1.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_1.export 2024-04-24T01:05:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:05:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T01:05:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_2 snap 2024-04-24T01:05:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:05:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:05:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:05:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-04-24T01:05:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:05:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-04-24T01:05:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:05:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:05:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:05:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:05:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-04-24T01:05:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:05:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:05:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:05:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:05:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-04-24T01:05:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:05:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:05:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:05:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:05:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:05:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:05:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-04-24T01:05:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:05:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:05:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:05:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-04-24T01:05:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 7 closest matches: 2024-04-24T01:05:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_22 2024-04-24T01:05:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_23 2024-04-24T01:05:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_24 2024-04-24T01:05:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_25 2024-04-24T01:05:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_26 2024-04-24T01:05:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_27 2024-04-24T01:05:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_28 2024-04-24T01:05:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-04-24T01:05:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:05:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:05:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:05:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-04-24T01:05:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:05:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_21 2024-04-24T01:05:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_29 2024-04-24T01:05:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:05:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:05:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-04-24T01:05:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:05:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:05:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:05:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:05:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:05:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:05:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:05:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-04-24T01:05:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:05:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:05:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_2 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:05:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_2 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_2 ']' 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:05:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:05:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:05:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:05:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-04-24T01:05:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:05:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-04-24T01:05:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 7 closest matches: 2024-04-24T01:05:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_22 2024-04-24T01:05:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_23 2024-04-24T01:05:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_24 2024-04-24T01:05:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_25 2024-04-24T01:05:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_26 2024-04-24T01:05:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_27 2024-04-24T01:05:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_28 2024-04-24T01:05:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_2 2024-04-24T01:05:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:05:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:05:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:05:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:05:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:05:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_2 2024-04-24T01:05:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:05:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_21 2024-04-24T01:05:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_29 2024-04-24T01:05:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_2 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_2 2024-04-24T01:05:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-04-24T01:05:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:05:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-04-24T01:05:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:05:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:05:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-24T01:05:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-24T01:05:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.status 2024-04-24T01:05:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-24T01:05:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.status 2024-04-24T01:05:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:05:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:05:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15751], [object_number=14, tag_tid=1, entry_tid=15750]]], state=connected] 2024-04-24T01:05:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15751], [object_number=14, tag_tid=1, entry_tid=15750]]], state=connected] 2024-04-24T01:05:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.status 2024-04-24T01:05:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-24T01:05:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-24T01:05:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.status 2024-04-24T01:05:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-24T01:05:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.status 2024-04-24T01:05:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:05:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:05:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15751], [object_number=14, tag_tid=1, entry_tid=15750]]], state=connected] 2024-04-24T01:05:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15751], [object_number=14, tag_tid=1, entry_tid=15750]]], state=connected] 2024-04-24T01:05:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.status 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:05:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:05:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-04-24T01:05:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_2': 2024-04-24T01:05:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:05:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:05:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:05:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b28f18245d 2024-04-24T01:05:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b28f18245d 2024-04-24T01:05:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:05:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:05:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:05:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:05:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:57:08 2024 2024-04-24T01:05:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:57:08 2024 2024-04-24T01:05:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:58:09 2024 2024-04-24T01:05:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:05:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10b28f18245d 2024-04-24T01:05:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:05:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:05:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 3fca205a-3d85-4256-bc77-901a3a091b41 2024-04-24T01:05:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:05:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_2 snap 2024-04-24T01:05:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.export 2024-04-24T01:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_2.export 2024-04-24T01:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_2.export 2024-04-24T01:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.export 2024-04-24T01:05:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:05:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_2.export 2024-04-24T01:05:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:05:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_2.export 2024-04-24T01:05:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_2.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_2.export 2024-04-24T01:05:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:05:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T01:05:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_3 snap 2024-04-24T01:05:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:05:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:05:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:05:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-04-24T01:05:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:05:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-04-24T01:05:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:05:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:05:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:05:57.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:05:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-04-24T01:05:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:05:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:05:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:05:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:05:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:05:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:05:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-04-24T01:05:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:05:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:05:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:05:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-04-24T01:05:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_32 2024-04-24T01:05:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_36 2024-04-24T01:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_37 2024-04-24T01:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:05:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:05:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-04-24T01:05:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_31 2024-04-24T01:05:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_35 2024-04-24T01:05:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_39 2024-04-24T01:05:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:05:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:05:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:05:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:05:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:05:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:05:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-04-24T01:05:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:05:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:05:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:05:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:05:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_3 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:05:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:05:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:05:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:05:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:05:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:05:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:05:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_3 2024-04-24T01:05:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_3 ']' 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:05:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:05:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-04-24T01:05:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_32 2024-04-24T01:05:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_36 2024-04-24T01:05:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_37 2024-04-24T01:05:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:05:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:05:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:05:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2024-04-24T01:05:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_31 2024-04-24T01:05:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_35 2024-04-24T01:05:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_39 2024-04-24T01:05:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_3 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_3 2024-04-24T01:05:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-24T01:05:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:05:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-24T01:05:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:05:59.051 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:59.050+0000 7fd951b0e640 -1 reset not still connected to 0x56372331d1e0 2024-04-24T01:05:59.051 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:59.050+0000 7fd951b0e640 -1 reset not still connected to 0x563727aaaf70 2024-04-24T01:05:59.051 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:59.050+0000 7fd951b0e640 -1 reset not still connected to 0x5637301e40d0 2024-04-24T01:05:59.051 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:59.050+0000 7fd951b0e640 -1 reset not still connected to 0x5637301ea0d0 2024-04-24T01:05:59.051 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:59.050+0000 7fd951b0e640 -1 reset not still connected to 0x5637301eaea0 2024-04-24T01:05:59.051 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:59.050+0000 7fd951b0e640 -1 reset not still connected to 0x5637301f75f0 2024-04-24T01:05:59.051 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:05:59.050+0000 7fd951b0e640 -1 reset not still connected to 0x563730230a90 2024-04-24T01:05:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:05:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:05:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:05:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=9, tag_tid=1, entry_tid=9789], [object_number=8, tag_tid=1, entry_tid=9788]]], state=connected] 2024-04-24T01:05:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:05:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:05:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-04-24T01:05:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:05:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-04-24T01:05:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-24T01:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-24T01:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:05:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-24T01:05:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:05:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:05:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:05:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:05:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:05:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=9, tag_tid=1, entry_tid=9789], [object_number=8, tag_tid=1, entry_tid=9788]]], state=connected] 2024-04-24T01:05:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:05:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:05:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:05:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=9791' '!=' '' 2024-04-24T01:05:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:05:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=9791' = '' 2024-04-24T01:05:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:05:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:05:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:05:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:05:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:05:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:05:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:05:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:05:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:05:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:05:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:05:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:05:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:05:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:05:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:05:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:05:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9791 2024-04-24T01:05:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 9791 2024-04-24T01:05:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:05:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:05:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_3 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_3 ']' 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:05:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:05:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:05:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:05:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:05:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:05:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:05:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:05:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:05:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:05:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:05:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:05:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:05:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:05:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:05:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-04-24T01:05:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_32 2024-04-24T01:05:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_36 2024-04-24T01:05:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_37 2024-04-24T01:05:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:05:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:05:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:05:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:05:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:05:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:05:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:05:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2024-04-24T01:05:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:05:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_31 2024-04-24T01:05:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_35 2024-04-24T01:05:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_39 2024-04-24T01:05:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:05:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:05:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_3 2024-04-24T01:05:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:05:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:05:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:05:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:05:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:05:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:05:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:05:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_3 2024-04-24T01:05:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:05:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-04-24T01:05:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:05:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-04-24T01:05:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:05:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:05:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:05:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:05:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-24T01:05:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-24T01:05:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:05:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:05:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-24T01:05:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:06:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=9, tag_tid=1, entry_tid=9789], [object_number=8, tag_tid=1, entry_tid=9788]]], state=connected] 2024-04-24T01:06:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-04-24T01:06:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-04-24T01:06:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-24T01:06:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-24T01:06:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-24T01:06:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:06:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=9, tag_tid=1, entry_tid=9789], [object_number=8, tag_tid=1, entry_tid=9788]]], state=connected] 2024-04-24T01:06:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:06:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:06:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=9791' '!=' 'object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:06:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:06:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_3 2024-04-24T01:06:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:06:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_3 ']' 2024-04-24T01:06:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-04-24T01:06:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-04-24T01:06:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-04-24T01:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_32 2024-04-24T01:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_36 2024-04-24T01:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_37 2024-04-24T01:06:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2024-04-24T01:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2024-04-24T01:06:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:06:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_31 2024-04-24T01:06:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_35 2024-04-24T01:06:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_39 2024-04-24T01:06:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_3 2024-04-24T01:06:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_3 2024-04-24T01:06:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-04-24T01:06:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-04-24T01:06:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-24T01:06:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-24T01:06:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-24T01:06:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:06:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=9, tag_tid=1, entry_tid=9789], [object_number=8, tag_tid=1, entry_tid=9788]]], state=connected] 2024-04-24T01:06:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-04-24T01:06:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-04-24T01:06:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-24T01:06:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-24T01:06:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-24T01:06:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:06:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=9, tag_tid=1, entry_tid=9789], [object_number=8, tag_tid=1, entry_tid=9788]]], state=connected] 2024-04-24T01:06:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:06:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:06:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=9791' '!=' 'object_number=11, tag_tid=1, entry_tid=9791' 2024-04-24T01:06:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:06:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_3 2024-04-24T01:06:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_3 ']' 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-04-24T01:06:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:06:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_32 2024-04-24T01:06:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_36 2024-04-24T01:06:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_37 2024-04-24T01:06:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2024-04-24T01:06:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 3 closest matches: 2024-04-24T01:06:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_31 2024-04-24T01:06:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_35 2024-04-24T01:06:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_39 2024-04-24T01:06:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_3 2024-04-24T01:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_3 2024-04-24T01:06:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-04-24T01:06:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-04-24T01:06:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-24T01:06:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-24T01:06:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-24T01:06:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:06:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:06:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-04-24T01:06:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-04-24T01:06:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:06:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-24T01:06:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-24T01:06:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-24T01:06:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:06:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9791], [object_number=10, tag_tid=1, entry_tid=9790]]], state=connected] 2024-04-24T01:06:02.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.status 2024-04-24T01:06:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-04-24T01:06:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:06:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:06:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-04-24T01:06:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_3': 2024-04-24T01:06:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:06:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:06:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 116871b0aada 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.116871b0aada 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:57:22 2024 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:57:22 2024 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:58:23 2024 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:06:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 116871b0aada 2024-04-24T01:06:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:06:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:06:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 954d7988-431e-4752-8a68-be7a95a93293 2024-04-24T01:06:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:06:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_3 snap 2024-04-24T01:06:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:06:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:06:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.export 2024-04-24T01:06:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_3.export 2024-04-24T01:06:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_3.export 2024-04-24T01:06:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.export 2024-04-24T01:06:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_3.export 2024-04-24T01:06:03.762 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:06:03.758+0000 7efd51cce640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:06:03.763 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:06:03.762+0000 7efd4bcc2640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-04-24T01:06:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_3.export 2024-04-24T01:06:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_3.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_3.export 2024-04-24T01:06:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:06:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T01:06:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_4 snap 2024-04-24T01:06:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:06:07.103 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:06:07.097+0000 7f63163ce640 -1 reset not still connected to 0x55c560aac340 2024-04-24T01:06:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-04-24T01:06:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-04-24T01:06:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:06:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:06:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:06:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-04-24T01:06:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:06:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:06:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:06:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:06:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:06:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:06:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-04-24T01:06:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:06:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:06:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-04-24T01:06:09.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:06:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_40 2024-04-24T01:06:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_41 2024-04-24T01:06:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-04-24T01:06:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:06:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:06:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:06:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-04-24T01:06:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:06:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_43 2024-04-24T01:06:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_44 2024-04-24T01:06:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-04-24T01:06:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:06:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:06:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-04-24T01:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:06:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:06:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_4 2024-04-24T01:06:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:06:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:06:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_4 2024-04-24T01:06:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_4 ']' 2024-04-24T01:06:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-04-24T01:06:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-04-24T01:06:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-04-24T01:06:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-04-24T01:06:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:06:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_40 2024-04-24T01:06:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_41 2024-04-24T01:06:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-04-24T01:06:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:06:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_43 2024-04-24T01:06:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_44 2024-04-24T01:06:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_4 2024-04-24T01:06:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_4 2024-04-24T01:06:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-04-24T01:06:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-04-24T01:06:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-24T01:06:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-24T01:06:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-24T01:06:10.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:10.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870]]], state=connected] 2024-04-24T01:06:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=9872], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870], [object_number=9, tag_tid=1, entry_tid=9869]]], state=connected] 2024-04-24T01:06:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-04-24T01:06:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-24T01:06:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-24T01:06:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.153 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:06:10.149+0000 7fd951b0e640 -1 reset not still connected to 0x56372331d1e0 2024-04-24T01:06:10.153 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:06:10.149+0000 7fd951b0e640 -1 reset not still connected to 0x563727aaaf70 2024-04-24T01:06:10.153 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:06:10.149+0000 7fd951b0e640 -1 reset not still connected to 0x5637301ea0d0 2024-04-24T01:06:10.153 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:06:10.149+0000 7fd951b0e640 -1 reset not still connected to 0x5637301eaea0 2024-04-24T01:06:10.153 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:06:10.149+0000 7fd951b0e640 -1 reset not still connected to 0x5637301f75f0 2024-04-24T01:06:10.156 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:06:10.153+0000 7fd951b0e640 -1 reset not still connected to 0x563730230a90 2024-04-24T01:06:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870]]], state=connected] 2024-04-24T01:06:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=9872], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870], [object_number=9, tag_tid=1, entry_tid=9869]]], state=connected] 2024-04-24T01:06:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=9872' 2024-04-24T01:06:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=9872' 2024-04-24T01:06:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=9872' '!=' '' 2024-04-24T01:06:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:06:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=9872' = '' 2024-04-24T01:06:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=9872' 2024-04-24T01:06:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:06:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=9872' 2024-04-24T01:06:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:06:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:06:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=9872' 2024-04-24T01:06:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:10.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9872 2024-04-24T01:06:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 9872 2024-04-24T01:06:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_4 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_4 ']' 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-04-24T01:06:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-04-24T01:06:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:06:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_40 2024-04-24T01:06:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_41 2024-04-24T01:06:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-04-24T01:06:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-04-24T01:06:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:06:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_43 2024-04-24T01:06:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_44 2024-04-24T01:06:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_4 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_4 2024-04-24T01:06:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-04-24T01:06:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-04-24T01:06:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-24T01:06:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-24T01:06:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-24T01:06:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870]]], state=connected] 2024-04-24T01:06:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=9872], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870], [object_number=9, tag_tid=1, entry_tid=9869]]], state=connected] 2024-04-24T01:06:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-04-24T01:06:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-04-24T01:06:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-24T01:06:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-24T01:06:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-24T01:06:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870]]], state=connected] 2024-04-24T01:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=9872], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870], [object_number=9, tag_tid=1, entry_tid=9869]]], state=connected] 2024-04-24T01:06:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=9872' 2024-04-24T01:06:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=9872' 2024-04-24T01:06:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=9872' '!=' 'object_number=8, tag_tid=1, entry_tid=9872' 2024-04-24T01:06:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:06:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_4 2024-04-24T01:06:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_4 ']' 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-04-24T01:06:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-04-24T01:06:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:06:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_40 2024-04-24T01:06:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_41 2024-04-24T01:06:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-04-24T01:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 2 closest matches: 2024-04-24T01:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_43 2024-04-24T01:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_44 2024-04-24T01:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_4 2024-04-24T01:06:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_4 2024-04-24T01:06:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-04-24T01:06:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-04-24T01:06:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-24T01:06:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-24T01:06:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-24T01:06:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870]]], state=connected] 2024-04-24T01:06:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870]]], state=connected] 2024-04-24T01:06:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-24T01:06:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-24T01:06:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-24T01:06:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870]]], state=connected] 2024-04-24T01:06:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9871], [object_number=10, tag_tid=1, entry_tid=9870]]], state=connected] 2024-04-24T01:06:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.status 2024-04-24T01:06:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-04-24T01:06:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:06:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-04-24T01:06:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_4': 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 11d959ada356 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.11d959ada356 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:57:32 2024 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:57:32 2024 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:58:33 2024 2024-04-24T01:06:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:06:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 11d959ada356 2024-04-24T01:06:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:06:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:06:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 5e43bfa8-db3a-4b1c-bb46-38e066dab2d8 2024-04-24T01:06:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:06:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_4 snap 2024-04-24T01:06:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:06:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:06:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.export 2024-04-24T01:06:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_4.export 2024-04-24T01:06:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_4.export 2024-04-24T01:06:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.export 2024-04-24T01:06:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_4.export 2024-04-24T01:06:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_4.export 2024-04-24T01:06:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_4.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_4.export 2024-04-24T01:06:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:06:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T01:06:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_5 snap 2024-04-24T01:06:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:06:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:06:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-04-24T01:06:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:06:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-04-24T01:06:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:06:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:06:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-04-24T01:06:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:06:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:06:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:06:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:06:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-04-24T01:06:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:06:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:06:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:06:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:06:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:06:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:06:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-04-24T01:06:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:06:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:06:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:06:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:06:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:06:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-04-24T01:06:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-24T01:06:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:06:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:06:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-04-24T01:06:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 1 closest matches: 2024-04-24T01:06:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status mirror/image_50 2024-04-24T01:06:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-04-24T01:06:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:06:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:06:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:06:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-04-24T01:06:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:06:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:06:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:06:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:06:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:06:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:06:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:06:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_5 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_5 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_5 ']' 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-04-24T01:06:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-04-24T01:06:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-24T01:06:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2024-04-24T01:06:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:no valid command found; 1 closest matches: 2024-04-24T01:06:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush mirror/image_50 2024-04-24T01:06:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_5 2024-04-24T01:06:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_5 2024-04-24T01:06:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-04-24T01:06:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:06:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-04-24T01:06:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:06:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-24T01:06:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-24T01:06:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.status 2024-04-24T01:06:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-24T01:06:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.status 2024-04-24T01:06:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.status 2024-04-24T01:06:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-04-24T01:06:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:06:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-04-24T01:06:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:06:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-24T01:06:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-24T01:06:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.status 2024-04-24T01:06:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-24T01:06:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.status 2024-04-24T01:06:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.status 2024-04-24T01:06:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-04-24T01:06:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:06:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:06:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-04-24T01:06:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_5': 2024-04-24T01:06:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:06:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:06:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b217f7a91 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b217f7a91 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:57:42 2024 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:57:42 2024 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:58:43 2024 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10b217f7a91 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:06:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 92ae8c17-e3b0-4c0e-ab2e-338866a10dcb 2024-04-24T01:06:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:06:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_5 snap 2024-04-24T01:06:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:06:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:06:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.export 2024-04-24T01:06:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_5.export 2024-04-24T01:06:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_5.export 2024-04-24T01:06:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.export 2024-04-24T01:06:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_5.export 2024-04-24T01:06:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_5.export 2024-04-24T01:06:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_5.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_5.export 2024-04-24T01:06:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:06:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T01:06:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_6 snap 2024-04-24T01:06:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-04-24T01:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-04-24T01:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:06:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:22.386 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:06:22.385+0000 7efd60937640 -1 reset not still connected to 0x559af4e14270 2024-04-24T01:06:22.387 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:06:22.385+0000 7efd60937640 -1 reset not still connected to 0x559af4e1fc70 2024-04-24T01:06:22.394 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:06:22.389+0000 7efd60937640 -1 reset not still connected to 0x559af8c6e680 2024-04-24T01:06:22.395 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:06:22.389+0000 7efd60937640 -1 reset not still connected to 0x559af8caea90 2024-04-24T01:06:22.395 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:06:22.389+0000 7efd60937640 -1 reset not still connected to 0x559b1296a5b0 2024-04-24T01:06:22.395 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:06:22.393+0000 7efd60937640 -1 reset not still connected to 0x559b1437aa90 2024-04-24T01:06:22.395 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:06:22.393+0000 7efd60937640 -1 reset not still connected to 0x559b144b0270 2024-04-24T01:06:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-04-24T01:06:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:06:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:06:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:06:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-04-24T01:06:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:06:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:06:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:06:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:06:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:06:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:06:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:06:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-04-24T01:06:22.962 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8158073710613057) > weight (0.25) 2024-04-24T01:06:22.963 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T01:06:22.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T01:06:22.963 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-24T01:06:22.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:06:22.965+0000 7f583afd3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 57180) UID: 1000 2024-04-24T01:06:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-24T01:06:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-04-24T01:06:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:06:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:06:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-04-24T01:06:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:06:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:06:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:06:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:06:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_6 2024-04-24T01:06:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:06:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:06:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_6 2024-04-24T01:06:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_6 ']' 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-24T01:06:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-24T01:06:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-24T01:06:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-24T01:06:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10329], [object_number=8, tag_tid=1, entry_tid=10328], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-24T01:06:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-24T01:06:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10329], [object_number=8, tag_tid=1, entry_tid=10328], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10329' '!=' '' 2024-04-24T01:06:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:06:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10329' = '' 2024-04-24T01:06:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:06:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:06:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10329 2024-04-24T01:06:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 10329 2024-04-24T01:06:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_6 2024-04-24T01:06:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_6 ']' 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-24T01:06:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-24T01:06:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-24T01:06:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-24T01:06:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-24T01:06:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10329], [object_number=8, tag_tid=1, entry_tid=10328], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-24T01:06:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-24T01:06:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10329], [object_number=8, tag_tid=1, entry_tid=10328], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10329' '!=' 'object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:06:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_6 2024-04-24T01:06:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_6 ']' 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-24T01:06:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-24T01:06:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-24T01:06:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-24T01:06:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10329], [object_number=8, tag_tid=1, entry_tid=10328], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-24T01:06:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-24T01:06:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10329], [object_number=8, tag_tid=1, entry_tid=10328], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10329' '!=' 'object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_6 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_6 ']' 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-24T01:06:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-24T01:06:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-24T01:06:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-24T01:06:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10329], [object_number=8, tag_tid=1, entry_tid=10328], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-24T01:06:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-24T01:06:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10329], [object_number=8, tag_tid=1, entry_tid=10328], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10329' '!=' 'object_number=9, tag_tid=1, entry_tid=10329' 2024-04-24T01:06:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:06:27.964 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T01:06:27.965 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:06:27.965 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T01:06:27.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T01:06:27.965 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T01:06:27.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T01:06:27.968 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-04-24T01:06:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_6 2024-04-24T01:06:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_6 ']' 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-24T01:06:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-24T01:06:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-24T01:06:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-24T01:06:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-24T01:06:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-24T01:06:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-24T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-24T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-24T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-24T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-24T01:06:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10327], [object_number=10, tag_tid=1, entry_tid=10326]]], state=connected] 2024-04-24T01:06:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.status 2024-04-24T01:06:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-04-24T01:06:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:06:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-04-24T01:06:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_6': 2024-04-24T01:06:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122de6176741 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122de6176741 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:57:54 2024 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:57:54 2024 2024-04-24T01:06:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:58:55 2024 2024-04-24T01:06:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:06:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122de6176741 2024-04-24T01:06:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:06:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:06:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e6bc7320-093c-4352-8f50-d4907de5e4f0 2024-04-24T01:06:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:06:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_6 snap 2024-04-24T01:06:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:06:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:06:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.export 2024-04-24T01:06:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_6.export 2024-04-24T01:06:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_6.export 2024-04-24T01:06:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.export 2024-04-24T01:06:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_6.export 2024-04-24T01:06:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_6.export 2024-04-24T01:06:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_6.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_6.export 2024-04-24T01:06:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:06:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T01:06:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_7 snap 2024-04-24T01:06:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:06:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-04-24T01:06:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-04-24T01:06:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:06:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:33.450 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7502684372594518) > weight (0.25) 2024-04-24T01:06:33.450 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5181909814741117) > weight (0.25) 2024-04-24T01:06:33.450 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7851491134621662) > weight (0.25) 2024-04-24T01:06:33.451 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.38644605883346694) > weight (0.25) 2024-04-24T01:06:33.451 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-04-24T01:06:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-04-24T01:06:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:06:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:06:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-04-24T01:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:06:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:06:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-04-24T01:06:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:06:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:06:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-04-24T01:06:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T01:06:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-04-24T01:06:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:06:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:06:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:06:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-04-24T01:06:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T01:06:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-04-24T01:06:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:06:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:06:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:06:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:34.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:34.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-04-24T01:06:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:06:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:06:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:06:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:06:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_7 2024-04-24T01:06:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:06:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:06:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_7 2024-04-24T01:06:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_7 ']' 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-04-24T01:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-04-24T01:06:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T01:06:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2024-04-24T01:06:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T01:06:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_7 2024-04-24T01:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_7 2024-04-24T01:06:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-04-24T01:06:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-04-24T01:06:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-24T01:06:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-24T01:06:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-24T01:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046]]], state=connected] 2024-04-24T01:06:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10048], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046], [object_number=9, tag_tid=1, entry_tid=10045]]], state=connected] 2024-04-24T01:06:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-04-24T01:06:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-04-24T01:06:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-24T01:06:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-24T01:06:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-24T01:06:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046]]], state=connected] 2024-04-24T01:06:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10048], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046], [object_number=9, tag_tid=1, entry_tid=10045]]], state=connected] 2024-04-24T01:06:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10048' 2024-04-24T01:06:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10048' 2024-04-24T01:06:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10048' '!=' '' 2024-04-24T01:06:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:06:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10048' = '' 2024-04-24T01:06:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=10048' 2024-04-24T01:06:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:06:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=10048' 2024-04-24T01:06:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:06:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=10048' 2024-04-24T01:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10048 2024-04-24T01:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 10048 2024-04-24T01:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_7 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_7 ']' 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-04-24T01:06:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-04-24T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2024-04-24T01:06:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T01:06:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_7 2024-04-24T01:06:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_7 2024-04-24T01:06:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-04-24T01:06:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-04-24T01:06:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-24T01:06:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-24T01:06:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-24T01:06:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046]]], state=connected] 2024-04-24T01:06:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10048], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046], [object_number=9, tag_tid=1, entry_tid=10045]]], state=connected] 2024-04-24T01:06:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-24T01:06:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-24T01:06:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-24T01:06:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046]]], state=connected] 2024-04-24T01:06:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10048], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046], [object_number=9, tag_tid=1, entry_tid=10045]]], state=connected] 2024-04-24T01:06:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10048' 2024-04-24T01:06:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10048' 2024-04-24T01:06:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10048' '!=' 'object_number=8, tag_tid=1, entry_tid=10048' 2024-04-24T01:06:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:06:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_7 2024-04-24T01:06:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_7 ']' 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-04-24T01:06:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-04-24T01:06:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T01:06:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2024-04-24T01:06:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T01:06:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_7 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_7 2024-04-24T01:06:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-04-24T01:06:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-04-24T01:06:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-24T01:06:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-24T01:06:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-24T01:06:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046]]], state=connected] 2024-04-24T01:06:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046]]], state=connected] 2024-04-24T01:06:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-04-24T01:06:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-04-24T01:06:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:06:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-24T01:06:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-24T01:06:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-24T01:06:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046]]], state=connected] 2024-04-24T01:06:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10047], [object_number=10, tag_tid=1, entry_tid=10046]]], state=connected] 2024-04-24T01:06:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.status 2024-04-24T01:06:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-04-24T01:06:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:06:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:37.130 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:06:37.129+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dade57930 2024-04-24T01:06:37.130 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:06:37.129+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dade5f790 2024-04-24T01:06:37.135 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:06:37.133+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dafbc3ad0 2024-04-24T01:06:37.135 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:06:37.133+0000 7fbc3ce8e640 -1 reset not still connected to 0x556db50816c0 2024-04-24T01:06:37.135 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:06:37.133+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de16c5450 2024-04-24T01:06:37.135 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:06:37.133+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de29052b0 2024-04-24T01:06:37.139 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:06:37.137+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de29184e0 2024-04-24T01:06:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-04-24T01:06:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_7': 2024-04-24T01:06:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:06:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:06:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:06:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d8bd5e65be 2024-04-24T01:06:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d8bd5e65be 2024-04-24T01:06:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:06:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:58:05 2024 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:58:05 2024 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:59:06 2024 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d8bd5e65be 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 9e57c636-7ff5-4c33-ab3a-285bc7478660 2024-04-24T01:06:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:06:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_7 snap 2024-04-24T01:06:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:06:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:06:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.export 2024-04-24T01:06:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_7.export 2024-04-24T01:06:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_7.export 2024-04-24T01:06:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.export 2024-04-24T01:06:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_7.export 2024-04-24T01:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_7.export 2024-04-24T01:06:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_7.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_7.export 2024-04-24T01:06:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:06:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T01:06:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_8 snap 2024-04-24T01:06:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:06:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-04-24T01:06:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-04-24T01:06:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:06:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-04-24T01:06:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:06:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:06:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:06:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-04-24T01:06:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:06:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:06:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:06:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:06:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:06:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:06:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-04-24T01:06:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:06:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:06:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:06:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-04-24T01:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:06:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:06:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-04-24T01:06:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:06:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2024-04-24T01:06:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:06:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:06:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:06:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:06:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:06:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:06:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:06:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2024-04-24T01:06:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:06:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:06:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:06:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:06:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:06:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:06:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:06:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_8 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_8 2024-04-24T01:06:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_8 ']' 2024-04-24T01:06:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-04-24T01:06:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-04-24T01:06:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_8 2024-04-24T01:06:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_8 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:06:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:06:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:06:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-04-24T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-04-24T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-24T01:06:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-24T01:06:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-24T01:06:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10944], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942], [object_number=9, tag_tid=1, entry_tid=10941]]], state=connected] 2024-04-24T01:06:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-24T01:06:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10944], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942], [object_number=9, tag_tid=1, entry_tid=10941]]], state=connected] 2024-04-24T01:06:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10944' '!=' '' 2024-04-24T01:06:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:06:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10944' = '' 2024-04-24T01:06:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:06:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:06:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:06:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10944 2024-04-24T01:06:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 10944 2024-04-24T01:06:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_8 2024-04-24T01:06:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_8 ']' 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-04-24T01:06:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:44.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:44.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:44.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_8 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_8 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:06:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:06:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:06:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:06:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:06:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:06:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-04-24T01:06:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-04-24T01:06:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-24T01:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-24T01:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-24T01:06:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10944], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942], [object_number=9, tag_tid=1, entry_tid=10941]]], state=connected] 2024-04-24T01:06:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-04-24T01:06:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-04-24T01:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-24T01:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-24T01:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-24T01:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10944], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942], [object_number=9, tag_tid=1, entry_tid=10941]]], state=connected] 2024-04-24T01:06:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10944' '!=' 'object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_8 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_8 ']' 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-04-24T01:06:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_8 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_8 2024-04-24T01:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:06:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:06:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:06:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:06:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:06:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-04-24T01:06:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-04-24T01:06:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-24T01:06:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-24T01:06:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-24T01:06:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10944], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942], [object_number=9, tag_tid=1, entry_tid=10941]]], state=connected] 2024-04-24T01:06:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-24T01:06:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-24T01:06:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10944], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942], [object_number=9, tag_tid=1, entry_tid=10941]]], state=connected] 2024-04-24T01:06:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10944' '!=' 'object_number=8, tag_tid=1, entry_tid=10944' 2024-04-24T01:06:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_8 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_8 ']' 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-04-24T01:06:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-04-24T01:06:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_8 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T01:06:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_8 2024-04-24T01:06:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:06:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:06:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:06:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:06:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:06:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:06:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:06:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:06:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_8 2024-04-24T01:06:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-04-24T01:06:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-04-24T01:06:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-24T01:06:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-24T01:06:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-24T01:06:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-04-24T01:06:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-04-24T01:06:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:06:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-24T01:06:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-24T01:06:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-24T01:06:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:06:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:06:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10943], [object_number=10, tag_tid=1, entry_tid=10942]]], state=connected] 2024-04-24T01:06:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.status 2024-04-24T01:06:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:06:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-04-24T01:06:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_8': 2024-04-24T01:06:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:06:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:06:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:06:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122d65d5aa86 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122d65d5aa86 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:58:15 2024 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:58:15 2024 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:59:16 2024 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122d65d5aa86 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:06:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 75cbacd9-1618-412e-a157-b680d23e5f14 2024-04-24T01:06:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:06:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_8 snap 2024-04-24T01:06:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:06:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:06:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.export 2024-04-24T01:06:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_8.export 2024-04-24T01:06:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_8.export 2024-04-24T01:06:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.export 2024-04-24T01:06:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_8.export 2024-04-24T01:06:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_8.export 2024-04-24T01:06:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_8.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_8.export 2024-04-24T01:06:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:06:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T01:06:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_9 snap 2024-04-24T01:06:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:06:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-04-24T01:06:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:06:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:06:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:06:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-04-24T01:06:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:06:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:06:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:06:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:06:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:06:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:06:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:06:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:06:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-04-24T01:06:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-04-24T01:06:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-04-24T01:06:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:06:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:06:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:06:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:06:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:06:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-04-24T01:06:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:06:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:06:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:06:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:06:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:06:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:06:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:06:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-04-24T01:06:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:06:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_9 2024-04-24T01:06:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:06:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:06:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_9 2024-04-24T01:06:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_9 ']' 2024-04-24T01:06:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_9' 2024-04-24T01:06:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2024-04-24T01:06:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2024-04-24T01:06:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2024-04-24T01:06:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-04-24T01:06:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_9 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_9 2024-04-24T01:06:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_9 2024-04-24T01:06:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:06:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_9 '' 2024-04-24T01:06:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:06:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-04-24T01:06:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_9.status 2024-04-24T01:06:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-04-24T01:06:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:06:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:06:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5755], [object_number=6, tag_tid=1, entry_tid=5754]]], state=connected] 2024-04-24T01:06:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=5756], [object_number=7, tag_tid=1, entry_tid=5755], [object_number=6, tag_tid=1, entry_tid=5754], [object_number=5, tag_tid=1, entry_tid=5753]]], state=connected] 2024-04-24T01:06:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:06:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_9.status 2024-04-24T01:06:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-04-24T01:06:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:06:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:06:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5755], [object_number=6, tag_tid=1, entry_tid=5754]]], state=connected] 2024-04-24T01:06:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=5756], [object_number=7, tag_tid=1, entry_tid=5755], [object_number=6, tag_tid=1, entry_tid=5754], [object_number=5, tag_tid=1, entry_tid=5753]]], state=connected] 2024-04-24T01:06:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=5756' 2024-04-24T01:06:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=5756' 2024-04-24T01:06:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=5756' '!=' '' 2024-04-24T01:06:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:06:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=5756' = '' 2024-04-24T01:06:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=4, tag_tid=1, entry_tid=5756' 2024-04-24T01:06:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:06:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:06:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=5756' 2024-04-24T01:06:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:06:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:06:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:06:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:06:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=5756' 2024-04-24T01:06:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:06:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:06:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5756 2024-04-24T01:06:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 5756 2024-04-24T01:06:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_9 2024-04-24T01:06:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_9 ']' 2024-04-24T01:06:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_9' 2024-04-24T01:06:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2024-04-24T01:06:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2024-04-24T01:06:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-04-24T01:06:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_9 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_9 2024-04-24T01:06:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_9 2024-04-24T01:06:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:06:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_9 '' 2024-04-24T01:06:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:06:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:06:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-04-24T01:06:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_9.status 2024-04-24T01:06:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-04-24T01:06:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:06:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:06:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5755], [object_number=6, tag_tid=1, entry_tid=5754]]], state=connected] 2024-04-24T01:06:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5755], [object_number=6, tag_tid=1, entry_tid=5754]]], state=connected] 2024-04-24T01:06:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:06:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_9.status 2024-04-24T01:06:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:06:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-04-24T01:06:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:06:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:06:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:06:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5755], [object_number=6, tag_tid=1, entry_tid=5754]]], state=connected] 2024-04-24T01:06:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5755], [object_number=6, tag_tid=1, entry_tid=5754]]], state=connected] 2024-04-24T01:06:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.status 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:06:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-04-24T01:06:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_9': 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b262385c07 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b262385c07 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:58:26 2024 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:58:26 2024 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:58:26 2024 2024-04-24T01:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10b262385c07 2024-04-24T01:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 042a7054-99dd-4132-bbbf-541a7c5a117b 2024-04-24T01:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:06:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_9 snap 2024-04-24T01:06:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:06:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:06:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.export 2024-04-24T01:06:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_9.export 2024-04-24T01:06:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_9.export 2024-04-24T01:06:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.export 2024-04-24T01:06:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_9.export 2024-04-24T01:06:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:06:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_9.export 2024-04-24T01:06:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_9.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_9.export 2024-04-24T01:06:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:06:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T01:06:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_10 snap 2024-04-24T01:06:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:06:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:06:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-04-24T01:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-04-24T01:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:06:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:06:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:06:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:06:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-04-24T01:06:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:06:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:06:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:06:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:06:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-04-24T01:06:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:06:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:06:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:06:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:06:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:06:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:06:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:06:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-04-24T01:06:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:06:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:06:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:06:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-04-24T01:06:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:06:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-04-24T01:06:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:06:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:06:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:06:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-04-24T01:06:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:06:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:06:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:06:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:06:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:06:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:06:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:06:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:06:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:06:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:06:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:06:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:06:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:06:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:06:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:06:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:06:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2024-04-24T01:06:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:06:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:06:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:06:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:06:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_10 2024-04-24T01:06:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:06:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:06:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:06:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:06:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:06:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:06:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:06:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:06:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_10 2024-04-24T01:06:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:06:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:06:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:06:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_10 ']' 2024-04-24T01:06:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-24T01:06:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-24T01:06:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:06:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:06:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:06:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:06:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:06:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:06:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:06:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:06:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-24T01:06:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:06:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:06:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:06:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:06:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:06:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:06:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-24T01:06:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:06:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:06:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:06:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:06:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:06:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:06:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:06:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-24T01:06:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:06:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:06:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_10 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:06:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:06:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:06:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:06:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:06:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:06:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:06:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:06:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:06:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:06:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:06:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:06:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:06:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:06:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:06:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:06:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:06:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:06:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:06:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_10 2024-04-24T01:07:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-24T01:07:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-24T01:07:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12030' '!=' '' 2024-04-24T01:07:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:07:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12030' = '' 2024-04-24T01:07:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:07:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:07:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:07:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12030 2024-04-24T01:07:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 12030 2024-04-24T01:07:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_10 2024-04-24T01:07:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:07:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_10 ']' 2024-04-24T01:07:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-24T01:07:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-24T01:07:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-24T01:07:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_10 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_10 2024-04-24T01:07:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-24T01:07:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-24T01:07:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-24T01:07:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-24T01:07:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12030' '!=' 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:07:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_10 2024-04-24T01:07:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:07:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_10 ']' 2024-04-24T01:07:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-24T01:07:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-24T01:07:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_10 2024-04-24T01:07:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_10 2024-04-24T01:07:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-24T01:07:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-24T01:07:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-24T01:07:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-24T01:07:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12030' '!=' 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:07:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_10 2024-04-24T01:07:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:07:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_10 ']' 2024-04-24T01:07:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-24T01:07:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-24T01:07:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-24T01:07:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_10 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_10 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-24T01:07:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-24T01:07:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12030' '!=' 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:07:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_10 2024-04-24T01:07:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:07:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_10 ']' 2024-04-24T01:07:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-24T01:07:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-24T01:07:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_10 2024-04-24T01:07:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_10 2024-04-24T01:07:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-24T01:07:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-24T01:07:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12030], [object_number=9, tag_tid=1, entry_tid=12029], [object_number=8, tag_tid=1, entry_tid=12028], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12030' '!=' 'object_number=10, tag_tid=1, entry_tid=12030' 2024-04-24T01:07:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:07:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_10 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_10 ']' 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_10 2024-04-24T01:07:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T01:07:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_10 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_10 2024-04-24T01:07:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-24T01:07:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-24T01:07:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-24T01:07:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-24T01:07:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:07:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-24T01:07:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-24T01:07:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-24T01:07:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12030], [object_number=11, tag_tid=1, entry_tid=12027]]], state=connected] 2024-04-24T01:07:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.status 2024-04-24T01:07:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-04-24T01:07:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:07:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:07:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-04-24T01:07:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_10': 2024-04-24T01:07:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:07:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:07:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:07:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d880fbd682 2024-04-24T01:07:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d880fbd682 2024-04-24T01:07:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:07:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:07:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:07:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:07:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:58:33 2024 2024-04-24T01:07:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:58:33 2024 2024-04-24T01:07:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:59:34 2024 2024-04-24T01:07:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:07:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d880fbd682 2024-04-24T01:07:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:07:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:07:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 37423567-5825-47e9-aae3-018898cca4ac 2024-04-24T01:07:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:07:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_10 snap 2024-04-24T01:07:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:07:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:07:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.export 2024-04-24T01:07:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_10.export 2024-04-24T01:07:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_10.export 2024-04-24T01:07:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.export 2024-04-24T01:07:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_10.export 2024-04-24T01:07:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_10.export 2024-04-24T01:07:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_10.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_10.export 2024-04-24T01:07:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:07:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_11 2024-04-24T01:07:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_11 snap 2024-04-24T01:07:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:07:11.956 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:07:11.956+0000 7f63163ce640 -1 reset not still connected to 0x55c5466f5790 2024-04-24T01:07:11.961 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:07:11.960+0000 7f63163ce640 -1 reset not still connected to 0x55c54dfbd860 2024-04-24T01:07:11.965 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:07:11.960+0000 7f63163ce640 -1 reset not still connected to 0x55c552de0340 2024-04-24T01:07:11.965 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:07:11.960+0000 7f63163ce640 -1 reset not still connected to 0x55c556bcb790 2024-04-24T01:07:11.966 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:07:11.960+0000 7f63163ce640 -1 reset not still connected to 0x55c566427450 2024-04-24T01:07:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-04-24T01:07:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-04-24T01:07:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:07:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-04-24T01:07:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:07:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:07:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:07:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-04-24T01:07:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:07:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:07:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:07:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:07:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:07:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:07:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:07:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:07:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-04-24T01:07:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-24T01:07:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-04-24T01:07:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:07:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:07:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-04-24T01:07:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:07:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:07:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:07:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:07:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:07:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:07:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_11 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:07:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:07:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_11 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_11 ']' 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-04-24T01:07:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-24T01:07:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_11 2024-04-24T01:07:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-04-24T01:07:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-04-24T01:07:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5194], [object_number=5, tag_tid=1, entry_tid=5193], [object_number=4, tag_tid=1, entry_tid=5192], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-04-24T01:07:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-04-24T01:07:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5194], [object_number=5, tag_tid=1, entry_tid=5193], [object_number=4, tag_tid=1, entry_tid=5192], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5194' '!=' '' 2024-04-24T01:07:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:07:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5194' = '' 2024-04-24T01:07:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:07:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:07:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:07:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5194 2024-04-24T01:07:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 5194 2024-04-24T01:07:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_11 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_11 ']' 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-24T01:07:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_11 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-04-24T01:07:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-04-24T01:07:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5194], [object_number=5, tag_tid=1, entry_tid=5193], [object_number=4, tag_tid=1, entry_tid=5192], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5194], [object_number=5, tag_tid=1, entry_tid=5193], [object_number=4, tag_tid=1, entry_tid=5192], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5194' '!=' 'object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:07:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_11 2024-04-24T01:07:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_11 ']' 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-24T01:07:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_11 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:16.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:16.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5194], [object_number=5, tag_tid=1, entry_tid=5193], [object_number=4, tag_tid=1, entry_tid=5192], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5194], [object_number=5, tag_tid=1, entry_tid=5193], [object_number=4, tag_tid=1, entry_tid=5192], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5194' '!=' 'object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:07:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_11 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_11 ']' 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-04-24T01:07:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-24T01:07:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_11 2024-04-24T01:07:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-04-24T01:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-04-24T01:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5194], [object_number=5, tag_tid=1, entry_tid=5193], [object_number=4, tag_tid=1, entry_tid=5192], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-04-24T01:07:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-04-24T01:07:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5194], [object_number=5, tag_tid=1, entry_tid=5193], [object_number=4, tag_tid=1, entry_tid=5192], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5194' '!=' 'object_number=6, tag_tid=1, entry_tid=5194' 2024-04-24T01:07:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_11 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_11 ']' 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-04-24T01:07:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-24T01:07:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_11 2024-04-24T01:07:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_11 2024-04-24T01:07:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-04-24T01:07:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-04-24T01:07:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-04-24T01:07:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-04-24T01:07:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:07:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-24T01:07:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-24T01:07:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-24T01:07:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5194], [object_number=7, tag_tid=1, entry_tid=5191]]], state=connected] 2024-04-24T01:07:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.status 2024-04-24T01:07:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:07:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-04-24T01:07:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_11': 2024-04-24T01:07:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:07:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:07:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:07:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1168e825ad4a 2024-04-24T01:07:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1168e825ad4a 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:58:46 2024 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:58:46 2024 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:58:46 2024 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 1168e825ad4a 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: d3171c2d-9f0b-4927-a4f0-874f45261881 2024-04-24T01:07:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:07:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_11 snap 2024-04-24T01:07:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:07:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:07:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.export 2024-04-24T01:07:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_11.export 2024-04-24T01:07:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_11.export 2024-04-24T01:07:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.export 2024-04-24T01:07:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_11.export 2024-04-24T01:07:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_11.export 2024-04-24T01:07:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_11.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_11.export 2024-04-24T01:07:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:07:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_12 2024-04-24T01:07:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_12 snap 2024-04-24T01:07:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:07:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-04-24T01:07:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:07:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:07:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:07:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-04-24T01:07:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:07:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:07:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:07:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:07:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:07:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:07:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:07:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:07:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:07:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-04-24T01:07:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:07:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:07:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-04-24T01:07:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:07:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:07:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:07:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:07:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-04-24T01:07:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2024-04-24T01:07:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:07:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2024-04-24T01:07:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:07:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:07:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:07:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:07:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:07:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_12 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:07:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:07:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_12 2024-04-24T01:07:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_12 ']' 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-24T01:07:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-24T01:07:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-24T01:07:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-24T01:07:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-24T01:07:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' '' 2024-04-24T01:07:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:07:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' = '' 2024-04-24T01:07:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:07:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:07:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:07:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:27.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9745 2024-04-24T01:07:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 9745 2024-04-24T01:07:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_12 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_12 ']' 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-24T01:07:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-24T01:07:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:07:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_12 2024-04-24T01:07:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_12 ']' 2024-04-24T01:07:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-24T01:07:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-24T01:07:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-24T01:07:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-24T01:07:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2024-04-24T01:07:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-24T01:07:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-24T01:07:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:07:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_12 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_12 ']' 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-24T01:07:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2024-04-24T01:07:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2024-04-24T01:07:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-24T01:07:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-24T01:07:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-24T01:07:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-24T01:07:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-24T01:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:07:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_12 2024-04-24T01:07:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_12 ']' 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_12 2024-04-24T01:07:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T01:07:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_12 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_12 2024-04-24T01:07:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-24T01:07:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-24T01:07:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-24T01:07:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-24T01:07:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-24T01:07:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-24T01:07:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.status 2024-04-24T01:07:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-04-24T01:07:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:07:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-04-24T01:07:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_12': 2024-04-24T01:07:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:07:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:07:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:07:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122d5249fa62 2024-04-24T01:07:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122d5249fa62 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:58:52 2024 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:58:52 2024 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 00:59:53 2024 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122d5249fa62 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 10ff2750-e07a-4278-885a-9078b880d567 2024-04-24T01:07:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:07:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_12 snap 2024-04-24T01:07:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:07:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:07:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.export 2024-04-24T01:07:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_12.export 2024-04-24T01:07:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_12.export 2024-04-24T01:07:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.export 2024-04-24T01:07:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_12.export 2024-04-24T01:07:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_12.export 2024-04-24T01:07:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_12.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_12.export 2024-04-24T01:07:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:07:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_13 2024-04-24T01:07:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_13 snap 2024-04-24T01:07:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:07:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-04-24T01:07:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:07:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:07:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:07:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:07:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-04-24T01:07:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:07:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:07:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:07:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:07:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:07:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:07:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-04-24T01:07:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:07:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:07:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:07:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-04-24T01:07:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-24T01:07:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:07:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:07:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-04-24T01:07:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:07:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:07:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:07:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:07:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:07:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:07:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_13 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:07:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:07:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:07:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_13 2024-04-24T01:07:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_13 ']' 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-04-24T01:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-24T01:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2024-04-24T01:07:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_13 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_13 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-24T01:07:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-24T01:07:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.status 2024-04-24T01:07:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-24T01:07:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.status 2024-04-24T01:07:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7899], [object_number=6, tag_tid=1, entry_tid=7898]]], state=connected] 2024-04-24T01:07:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7899], [object_number=6, tag_tid=1, entry_tid=7898]]], state=connected] 2024-04-24T01:07:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.status 2024-04-24T01:07:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-04-24T01:07:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_13 2024-04-24T01:07:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-04-24T01:07:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_13 2024-04-24T01:07:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-24T01:07:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-24T01:07:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.status 2024-04-24T01:07:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-24T01:07:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.status 2024-04-24T01:07:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7899], [object_number=6, tag_tid=1, entry_tid=7898]]], state=connected] 2024-04-24T01:07:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7899], [object_number=6, tag_tid=1, entry_tid=7898]]], state=connected] 2024-04-24T01:07:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.status 2024-04-24T01:07:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:07:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-04-24T01:07:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:07:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:07:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-04-24T01:07:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_13': 2024-04-24T01:07:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:07:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:07:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:07:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b2ba98bd87 2024-04-24T01:07:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b2ba98bd87 2024-04-24T01:07:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:59:03 2024 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:59:03 2024 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:00:04 2024 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10b2ba98bd87 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 48bf23f9-c1c6-448a-87b2-c12ae7488088 2024-04-24T01:07:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:07:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_13 snap 2024-04-24T01:07:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:07:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:07:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.export 2024-04-24T01:07:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_13.export 2024-04-24T01:07:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_13.export 2024-04-24T01:07:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.export 2024-04-24T01:07:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_13.export 2024-04-24T01:07:41.293 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:07:41.288+0000 7efd52cd0640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:07:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_13.export 2024-04-24T01:07:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_13.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_13.export 2024-04-24T01:07:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:07:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_14 2024-04-24T01:07:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_14 snap 2024-04-24T01:07:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-04-24T01:07:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:07:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:07:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:07:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-04-24T01:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:07:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:07:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-04-24T01:07:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:07:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:07:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-04-24T01:07:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-04-24T01:07:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:07:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:07:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:07:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-04-24T01:07:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2024-04-24T01:07:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:07:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:07:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:07:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2024-04-24T01:07:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:07:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:07:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:07:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:07:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:07:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:07:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:07:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-04-24T01:07:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_14 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_14 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_14 ']' 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-04-24T01:07:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-04-24T01:07:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_14 2024-04-24T01:07:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_14 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-04-24T01:07:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-04-24T01:07:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-24T01:07:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-24T01:07:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-24T01:07:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410]]], state=connected] 2024-04-24T01:07:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410], [object_number=9, tag_tid=1, entry_tid=11409], [object_number=8, tag_tid=1, entry_tid=11408]]], state=connected] 2024-04-24T01:07:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-24T01:07:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-24T01:07:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410]]], state=connected] 2024-04-24T01:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410], [object_number=9, tag_tid=1, entry_tid=11409], [object_number=8, tag_tid=1, entry_tid=11408]]], state=connected] 2024-04-24T01:07:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=11411' 2024-04-24T01:07:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=11411' 2024-04-24T01:07:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=11411' '!=' '' 2024-04-24T01:07:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:07:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=11411' = '' 2024-04-24T01:07:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=1, entry_tid=11411' 2024-04-24T01:07:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:07:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=11411' 2024-04-24T01:07:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:07:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:07:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=11411' 2024-04-24T01:07:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11411 2024-04-24T01:07:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 11411 2024-04-24T01:07:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_14 2024-04-24T01:07:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_14 ']' 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-04-24T01:07:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_14 2024-04-24T01:07:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_14 2024-04-24T01:07:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:46.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-04-24T01:07:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-04-24T01:07:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-24T01:07:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-24T01:07:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-24T01:07:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410]]], state=connected] 2024-04-24T01:07:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410], [object_number=9, tag_tid=1, entry_tid=11409], [object_number=8, tag_tid=1, entry_tid=11408]]], state=connected] 2024-04-24T01:07:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-04-24T01:07:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-04-24T01:07:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-24T01:07:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-24T01:07:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-24T01:07:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410]]], state=connected] 2024-04-24T01:07:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410], [object_number=9, tag_tid=1, entry_tid=11409], [object_number=8, tag_tid=1, entry_tid=11408]]], state=connected] 2024-04-24T01:07:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=11411' 2024-04-24T01:07:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=11411' 2024-04-24T01:07:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=11411' '!=' 'object_number=11, tag_tid=1, entry_tid=11411' 2024-04-24T01:07:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:07:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_14 2024-04-24T01:07:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_14 ']' 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_14 2024-04-24T01:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T01:07:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_14 2024-04-24T01:07:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:07:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:07:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:07:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:07:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:07:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:07:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_14 2024-04-24T01:07:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-04-24T01:07:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-04-24T01:07:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-24T01:07:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-24T01:07:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-24T01:07:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410]]], state=connected] 2024-04-24T01:07:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410]]], state=connected] 2024-04-24T01:07:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-04-24T01:07:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-04-24T01:07:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:07:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-24T01:07:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-24T01:07:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-24T01:07:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:07:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:07:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410]]], state=connected] 2024-04-24T01:07:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11411], [object_number=10, tag_tid=1, entry_tid=11410]]], state=connected] 2024-04-24T01:07:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.status 2024-04-24T01:07:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:07:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-04-24T01:07:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:07:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-04-24T01:07:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_14': 2024-04-24T01:07:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:07:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:07:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:07:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d8f4db9094 2024-04-24T01:07:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d8f4db9094 2024-04-24T01:07:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:07:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:07:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:59:12 2024 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:59:12 2024 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:00:13 2024 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d8f4db9094 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 9ccc1b62-e751-4cf7-bee0-6baa59442781 2024-04-24T01:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_14 snap 2024-04-24T01:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.export 2024-04-24T01:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_14.export 2024-04-24T01:07:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_14.export 2024-04-24T01:07:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.export 2024-04-24T01:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_14.export 2024-04-24T01:07:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_14.export 2024-04-24T01:07:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_14.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_14.export 2024-04-24T01:07:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:07:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_15 2024-04-24T01:07:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_15 snap 2024-04-24T01:07:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:07:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-04-24T01:07:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-04-24T01:07:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:07:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-04-24T01:07:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:07:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:07:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:07:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:07:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-04-24T01:07:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:07:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:07:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:07:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:07:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:07:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:07:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:07:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-04-24T01:07:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-04-24T01:07:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:07:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:07:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:07:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:07:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:07:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-04-24T01:07:51.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:51.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:07:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2024-04-24T01:07:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:07:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:07:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:07:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:07:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:07:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:07:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:07:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:07:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:07:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:07:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2024-04-24T01:07:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:07:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:07:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:07:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:07:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:07:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:07:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:07:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-04-24T01:07:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_15 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_15 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_15 ']' 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-24T01:07:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-24T01:07:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_15 2024-04-24T01:07:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:51.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-24T01:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-24T01:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=4, entry_tid=1' 2024-04-24T01:07:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-24T01:07:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=4, entry_tid=1' -a 'object_number=5, tag_tid=4, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=3, entry_tid=1' '!=' '' 2024-04-24T01:07:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:07:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=3, entry_tid=1' = '' 2024-04-24T01:07:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=4, entry_tid=1' 2024-04-24T01:07:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=4 2024-04-24T01:07:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:07:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=3 2024-04-24T01:07:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=4, entry_tid=1' 2024-04-24T01:07:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:07:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:07:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:07:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=1 2024-04-24T01:07:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 4 = 3 -a 1 -le 1 2024-04-24T01:07:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:07:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:07:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_15 2024-04-24T01:07:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_15 ']' 2024-04-24T01:07:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-24T01:07:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-24T01:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-24T01:07:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_15 2024-04-24T01:07:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:52.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=4, entry_tid=1' 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=4, entry_tid=1' -a 'object_number=5, tag_tid=4, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=3, entry_tid=1' '!=' 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:07:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_15 2024-04-24T01:07:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_15 ']' 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-24T01:07:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-24T01:07:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_15 2024-04-24T01:07:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-24T01:07:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-24T01:07:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=4, entry_tid=1' 2024-04-24T01:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-24T01:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-24T01:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=4, entry_tid=1' -a 'object_number=5, tag_tid=4, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=3, entry_tid=1' '!=' 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:07:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_15 2024-04-24T01:07:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_15 ']' 2024-04-24T01:07:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-24T01:07:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-24T01:07:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-24T01:07:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_15 2024-04-24T01:07:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-24T01:07:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-24T01:07:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=4, entry_tid=1' 2024-04-24T01:07:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-24T01:07:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=4, entry_tid=1' -a 'object_number=5, tag_tid=4, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=3, entry_tid=1' '!=' 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:07:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:07:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:07:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_15 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_15 ']' 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:07:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:07:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:07:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:07:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:07:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-24T01:07:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:07:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:07:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:07:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:07:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:07:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:07:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:07:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:07:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:07:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:07:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:07:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:07:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:07:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-24T01:07:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:07:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:07:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:07:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:07:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:07:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T01:07:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:07:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:07:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:07:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_15 2024-04-24T01:07:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:07:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:07:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:07:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:07:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:07:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:07:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:07:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:07:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_15 2024-04-24T01:07:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-24T01:07:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-24T01:07:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:07:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=4, entry_tid=1' 2024-04-24T01:07:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-24T01:07:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-24T01:07:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:07:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:07:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:07:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:07:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-24T01:07:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-24T01:07:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:07:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-24T01:07:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:07:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:07:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:07:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=4, entry_tid=1], [object_number=4, tag_tid=4, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:07:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.status 2024-04-24T01:07:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=4, entry_tid=1' 2024-04-24T01:07:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=4, entry_tid=1' -a 'object_number=5, tag_tid=4, entry_tid=1' = 'object_number=5, tag_tid=4, entry_tid=1' 2024-04-24T01:07:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-04-24T01:07:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:07:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:07:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:07:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:07:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_15': 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1168f84c6895 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1168f84c6895 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:59:27 2024 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:59:27 2024 2024-04-24T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:00:31 2024 2024-04-24T01:07:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:07:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 1168f84c6895 2024-04-24T01:07:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:07:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:07:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: fc6edd80-fe19-4761-8511-0b207141c6b4 2024-04-24T01:07:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_15 snap 2024-04-24T01:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.export 2024-04-24T01:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_15.export 2024-04-24T01:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_15.export 2024-04-24T01:07:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.export 2024-04-24T01:07:58.647 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:07:58.647+0000 7f6306f64640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-04-24T01:07:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_15.export 2024-04-24T01:07:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_15.export 2024-04-24T01:07:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_15.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_15.export 2024-04-24T01:07:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:07:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:07:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_16 2024-04-24T01:07:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_16 snap 2024-04-24T01:07:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:07:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:07:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:07:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:07:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:08:01.452 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6601700283175802) > weight (0.25) 2024-04-24T01:08:01.452 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7729875510008117) > weight (0.25) 2024-04-24T01:08:01.452 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9503485345857061) > weight (0.25) 2024-04-24T01:08:01.452 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.38479190824279297) > weight (0.25) 2024-04-24T01:08:01.452 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-04-24T01:08:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-04-24T01:08:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:08:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:08:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:08:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-04-24T01:08:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:08:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:08:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:08:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:08:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:08:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:08:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:08:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-04-24T01:08:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:08:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:08:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-04-24T01:08:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-04-24T01:08:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-04-24T01:08:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-04-24T01:08:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:08:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:08:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:08:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:08:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:08:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-04-24T01:08:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:08:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:08:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:08:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:08:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_16 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:08:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:08:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_16 2024-04-24T01:08:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_16 ']' 2024-04-24T01:08:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-04-24T01:08:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-04-24T01:08:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-04-24T01:08:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-04-24T01:08:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_16 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_16 2024-04-24T01:08:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:08:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:08:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:08:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:08:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:08:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-04-24T01:08:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-04-24T01:08:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-24T01:08:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-24T01:08:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-24T01:08:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254]]], state=connected] 2024-04-24T01:08:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11256], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254], [object_number=9, tag_tid=1, entry_tid=11253]]], state=connected] 2024-04-24T01:08:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-24T01:08:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-24T01:08:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254]]], state=connected] 2024-04-24T01:08:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11256], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254], [object_number=9, tag_tid=1, entry_tid=11253]]], state=connected] 2024-04-24T01:08:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11256' 2024-04-24T01:08:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11256' 2024-04-24T01:08:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11256' '!=' '' 2024-04-24T01:08:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:08:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11256' = '' 2024-04-24T01:08:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=11256' 2024-04-24T01:08:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:08:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=11256' 2024-04-24T01:08:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:08:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:08:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=11256' 2024-04-24T01:08:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=11256 2024-04-24T01:08:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 11256 2024-04-24T01:08:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_16 2024-04-24T01:08:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_16 ']' 2024-04-24T01:08:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-04-24T01:08:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-04-24T01:08:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-04-24T01:08:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_16 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_16 2024-04-24T01:08:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:08:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-04-24T01:08:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-04-24T01:08:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-24T01:08:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-24T01:08:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-24T01:08:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254]]], state=connected] 2024-04-24T01:08:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11256], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254], [object_number=9, tag_tid=1, entry_tid=11253]]], state=connected] 2024-04-24T01:08:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-04-24T01:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-04-24T01:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-24T01:08:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-24T01:08:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-24T01:08:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254]]], state=connected] 2024-04-24T01:08:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11256], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254], [object_number=9, tag_tid=1, entry_tid=11253]]], state=connected] 2024-04-24T01:08:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11256' 2024-04-24T01:08:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11256' 2024-04-24T01:08:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11256' '!=' 'object_number=8, tag_tid=1, entry_tid=11256' 2024-04-24T01:08:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:08:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_16 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_16 ']' 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-04-24T01:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_16 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T01:08:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_16 2024-04-24T01:08:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:08:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:08:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_16 2024-04-24T01:08:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-04-24T01:08:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-04-24T01:08:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-24T01:08:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-24T01:08:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-24T01:08:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254]]], state=connected] 2024-04-24T01:08:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254]]], state=connected] 2024-04-24T01:08:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-04-24T01:08:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-24T01:08:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-24T01:08:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254]]], state=connected] 2024-04-24T01:08:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11255], [object_number=10, tag_tid=1, entry_tid=11254]]], state=connected] 2024-04-24T01:08:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.status 2024-04-24T01:08:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-04-24T01:08:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:08:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-04-24T01:08:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_16': 2024-04-24T01:08:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:08:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122d16e4ed8c 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122d16e4ed8c 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:59:32 2024 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:59:32 2024 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:00:33 2024 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122d16e4ed8c 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 64087551-8c32-44d4-9fd5-d0f511237e58 2024-04-24T01:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:08:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_16 snap 2024-04-24T01:08:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:08:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:08:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.export 2024-04-24T01:08:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_16.export 2024-04-24T01:08:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_16.export 2024-04-24T01:08:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.export 2024-04-24T01:08:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_16.export 2024-04-24T01:08:09.245 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:08:09.243+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:08:09.245 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:08:09.243+0000 7efd51cce640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:08:09.246 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:08:09.247+0000 7efd524cf640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:08:09.246 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:08:09.247+0000 7efd4c4c3640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:08:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_16.export 2024-04-24T01:08:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_16.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_16.export 2024-04-24T01:08:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:08:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_17 2024-04-24T01:08:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_17 snap 2024-04-24T01:08:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:08:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-04-24T01:08:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-04-24T01:08:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:08:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-04-24T01:08:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:08:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:08:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:08:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-04-24T01:08:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:08:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:08:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:08:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:08:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:08:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:08:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-04-24T01:08:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:08:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:08:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:08:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:08:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:08:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-04-24T01:08:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-24T01:08:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:12.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-04-24T01:08:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-24T01:08:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:08:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:08:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-04-24T01:08:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:08:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:08:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:08:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:08:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:08:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_17 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:08:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:08:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_17 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_17 ']' 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-04-24T01:08:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-04-24T01:08:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-24T01:08:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2024-04-24T01:08:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2024-04-24T01:08:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-24T01:08:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_17 2024-04-24T01:08:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_17 2024-04-24T01:08:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-04-24T01:08:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:08:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-04-24T01:08:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:08:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-24T01:08:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-24T01:08:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-24T01:08:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:08:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:08:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15599], [object_number=14, tag_tid=1, entry_tid=15598]]], state=connected] 2024-04-24T01:08:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15600], [object_number=15, tag_tid=1, entry_tid=15599], [object_number=14, tag_tid=1, entry_tid=15598], [object_number=13, tag_tid=1, entry_tid=15597]]], state=connected] 2024-04-24T01:08:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:08:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-24T01:08:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-24T01:08:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:08:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:08:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15599], [object_number=14, tag_tid=1, entry_tid=15598]]], state=connected] 2024-04-24T01:08:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=15600], [object_number=15, tag_tid=1, entry_tid=15599], [object_number=14, tag_tid=1, entry_tid=15598], [object_number=13, tag_tid=1, entry_tid=15597]]], state=connected] 2024-04-24T01:08:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=15600' 2024-04-24T01:08:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=15600' 2024-04-24T01:08:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=15600' '!=' '' 2024-04-24T01:08:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:08:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=15600' = '' 2024-04-24T01:08:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=12, tag_tid=1, entry_tid=15600' 2024-04-24T01:08:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:08:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:08:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=15600' 2024-04-24T01:08:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:08:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:08:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:08:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=15600' 2024-04-24T01:08:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=15600 2024-04-24T01:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 15600 2024-04-24T01:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_17 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_17 ']' 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-04-24T01:08:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-04-24T01:08:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-24T01:08:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2024-04-24T01:08:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2024-04-24T01:08:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-24T01:08:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_17 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_17 2024-04-24T01:08:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-04-24T01:08:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:08:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-04-24T01:08:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:08:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-24T01:08:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-24T01:08:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-24T01:08:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:08:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:08:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15599], [object_number=14, tag_tid=1, entry_tid=15598]]], state=connected] 2024-04-24T01:08:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15599], [object_number=14, tag_tid=1, entry_tid=15598]]], state=connected] 2024-04-24T01:08:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:08:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-04-24T01:08:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:08:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-04-24T01:08:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:08:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-24T01:08:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-24T01:08:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-24T01:08:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:08:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:08:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15599], [object_number=14, tag_tid=1, entry_tid=15598]]], state=connected] 2024-04-24T01:08:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=15599], [object_number=14, tag_tid=1, entry_tid=15598]]], state=connected] 2024-04-24T01:08:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.status 2024-04-24T01:08:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:08:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:08:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-04-24T01:08:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:08:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:13.969 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T01:08:13.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T01:08:13.969 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-24T01:08:13.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:08:13.966+0000 7f4936ece640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 99524) UID: 1000 2024-04-24T01:08:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-04-24T01:08:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_17': 2024-04-24T01:08:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:08:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:08:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:08:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b25e528c84 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b25e528c84 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:59:44 2024 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:59:44 2024 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:00:45 2024 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10b25e528c84 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:08:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 141df2c4-aa92-4904-8f6e-93151024c34c 2024-04-24T01:08:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:08:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_17 snap 2024-04-24T01:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.export 2024-04-24T01:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_17.export 2024-04-24T01:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_17.export 2024-04-24T01:08:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.export 2024-04-24T01:08:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_17.export 2024-04-24T01:08:15.411 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:08:15.411+0000 7efd4e4c7640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:08:15.411 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:08:15.411+0000 7efd51cce640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:08:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_17.export 2024-04-24T01:08:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_17.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_17.export 2024-04-24T01:08:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:08:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_18 2024-04-24T01:08:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_18 snap 2024-04-24T01:08:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:08:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:08:15.702 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:08:15.698+0000 7f6305761640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 14 2024-04-24T01:08:15.704 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:08:15.702+0000 7f6303f5e640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 18 2024-04-24T01:08:15.706 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:08:15.702+0000 7f630375d640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 14 2024-04-24T01:08:15.970 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T01:08:15.970 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:08:15.971 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T01:08:15.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T01:08:15.971 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T01:08:15.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T01:08:15.974 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-04-24T01:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-04-24T01:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-04-24T01:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:08:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:08:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-04-24T01:08:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:08:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:08:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:08:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:08:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-04-24T01:08:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:08:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:08:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:08:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:08:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:08:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:08:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-04-24T01:08:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:08:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:08:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:08:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-04-24T01:08:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-24T01:08:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-04-24T01:08:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:08:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:08:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:08:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-04-24T01:08:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-24T01:08:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:08:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:08:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-04-24T01:08:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-24T01:08:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2024-04-24T01:08:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:08:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:08:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:08:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2024-04-24T01:08:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:08:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:08:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:08:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:08:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:08:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:08:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_18 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:08:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_18 2024-04-24T01:08:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_18 ']' 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-04-24T01:08:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-24T01:08:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-04-24T01:08:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-24T01:08:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_18 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_18 2024-04-24T01:08:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-24T01:08:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_18 2024-04-24T01:08:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:08:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:08:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_18 2024-04-24T01:08:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-04-24T01:08:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_18 2024-04-24T01:08:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-04-24T01:08:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_18 2024-04-24T01:08:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-24T01:08:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-24T01:08:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.status 2024-04-24T01:08:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-24T01:08:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.status 2024-04-24T01:08:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:08:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:08:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.status 2024-04-24T01:08:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:08:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_18 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_18 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-24T01:08:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.status 2024-04-24T01:08:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-24T01:08:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.status 2024-04-24T01:08:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:08:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:08:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.status 2024-04-24T01:08:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:08:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:08:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-04-24T01:08:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:08:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:08:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-04-24T01:08:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_18': 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d8e338043b 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d8e338043b 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 00:59:58 2024 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 00:59:58 2024 2024-04-24T01:08:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:00:59 2024 2024-04-24T01:08:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:08:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d8e338043b 2024-04-24T01:08:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:08:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:08:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: ade9577c-c44a-4c5c-9eca-0d4478f2e2a7 2024-04-24T01:08:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:08:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_18 snap 2024-04-24T01:08:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:08:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:08:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.export 2024-04-24T01:08:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_18.export 2024-04-24T01:08:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_18.export 2024-04-24T01:08:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.export 2024-04-24T01:08:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_18.export 2024-04-24T01:08:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_18.export 2024-04-24T01:08:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_18.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_18.export 2024-04-24T01:08:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_19 2024-04-24T01:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_19 snap 2024-04-24T01:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-04-24T01:08:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:08:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:08:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:08:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-04-24T01:08:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:08:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:08:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:08:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:08:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:08:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:08:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-04-24T01:08:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:08:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:08:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:08:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:08:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:08:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-04-24T01:08:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-24T01:08:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:08:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:08:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-04-24T01:08:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:08:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:08:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:08:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:08:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:08:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:08:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:08:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-04-24T01:08:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:08:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_19 2024-04-24T01:08:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:08:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:08:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_19 2024-04-24T01:08:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_19 ']' 2024-04-24T01:08:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-04-24T01:08:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-04-24T01:08:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-04-24T01:08:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-04-24T01:08:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-24T01:08:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2024-04-24T01:08:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2024-04-24T01:08:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-04-24T01:08:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-04-24T01:08:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-24T01:08:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-24T01:08:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-24T01:08:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:08:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:08:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-24T01:08:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:08:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-04-24T01:08:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-04-24T01:08:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-24T01:08:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-24T01:08:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-24T01:08:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:08:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:08:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-24T01:08:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=7, tag_tid=2, entry_tid=3' 2024-04-24T01:08:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-24T01:08:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=7, tag_tid=2, entry_tid=3' '!=' '' 2024-04-24T01:08:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:08:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=7, tag_tid=2, entry_tid=3' = '' 2024-04-24T01:08:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=7, tag_tid=2, entry_tid=3' 2024-04-24T01:08:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:08:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:08:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-24T01:08:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T01:08:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:08:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:08:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-24T01:08:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3 2024-04-24T01:08:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 3 2024-04-24T01:08:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_19 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_19 ']' 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-04-24T01:08:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-24T01:08:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2024-04-24T01:08:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2024-04-24T01:08:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-04-24T01:08:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-04-24T01:08:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-24T01:08:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-24T01:08:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-24T01:08:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:08:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:08:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-24T01:08:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:08:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-04-24T01:08:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-04-24T01:08:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-24T01:08:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-24T01:08:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:23.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:23.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-24T01:08:23.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:08:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:08:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-24T01:08:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=7, tag_tid=2, entry_tid=3' 2024-04-24T01:08:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-24T01:08:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=7, tag_tid=2, entry_tid=3' '!=' 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-24T01:08:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:08:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_19 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_19 ']' 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-04-24T01:08:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-24T01:08:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2024-04-24T01:08:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-04-24T01:08:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-04-24T01:08:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-24T01:08:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-24T01:08:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-24T01:08:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:08:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:08:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-24T01:08:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-24T01:08:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-24T01:08:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:08:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:08:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:08:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.status 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:08:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-04-24T01:08:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_19': 2024-04-24T01:08:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:08:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:08:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:08:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122df0925d4e 2024-04-24T01:08:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122df0925d4e 2024-04-24T01:08:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:08:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:08:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:00:37 2024 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:00:37 2024 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:01:44 2024 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122df0925d4e 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: c2253ea3-403c-4fe4-bda9-e9f48408dfe0 2024-04-24T01:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:08:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_19 snap 2024-04-24T01:08:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:08:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:08:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.export 2024-04-24T01:08:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_19.export 2024-04-24T01:08:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_19.export 2024-04-24T01:08:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.export 2024-04-24T01:08:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_19.export 2024-04-24T01:08:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_19.export 2024-04-24T01:08:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_19.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_19.export 2024-04-24T01:08:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:08:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_20 2024-04-24T01:08:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_20 snap 2024-04-24T01:08:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:08:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-04-24T01:08:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:08:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:08:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:08:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-04-24T01:08:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:08:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:08:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:08:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:08:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:08:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:08:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:08:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-04-24T01:08:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-04-24T01:08:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:08:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:08:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:08:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-04-24T01:08:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-04-24T01:08:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:08:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:08:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:08:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-04-24T01:08:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:08:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:08:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-04-24T01:08:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:08:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:08:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:08:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:08:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_20 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:08:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:08:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_20 2024-04-24T01:08:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_20 ']' 2024-04-24T01:08:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-24T01:08:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-24T01:08:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-24T01:08:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_20 2024-04-24T01:08:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_20 2024-04-24T01:08:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:08:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:08:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:08:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:08:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-24T01:08:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-24T01:08:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=9862], [object_number=9, tag_tid=1, entry_tid=9861], [object_number=8, tag_tid=1, entry_tid=9860], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-24T01:08:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=9862], [object_number=9, tag_tid=1, entry_tid=9861], [object_number=8, tag_tid=1, entry_tid=9860], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=9862' '!=' '' 2024-04-24T01:08:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:08:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=9862' = '' 2024-04-24T01:08:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:08:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:08:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:08:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9862 2024-04-24T01:08:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 9862 2024-04-24T01:08:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_20 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_20 ']' 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-24T01:08:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-24T01:08:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_20 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_20 2024-04-24T01:08:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:08:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:08:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:08:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:08:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-24T01:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-24T01:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=9862], [object_number=9, tag_tid=1, entry_tid=9861], [object_number=8, tag_tid=1, entry_tid=9860], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-24T01:08:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-24T01:08:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=9862], [object_number=9, tag_tid=1, entry_tid=9861], [object_number=8, tag_tid=1, entry_tid=9860], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=9862' '!=' 'object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_20 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_20 ']' 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-24T01:08:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:33.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_20 2024-04-24T01:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_20 2024-04-24T01:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-24T01:08:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-24T01:08:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=9862], [object_number=9, tag_tid=1, entry_tid=9861], [object_number=8, tag_tid=1, entry_tid=9860], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-24T01:08:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-24T01:08:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=9862], [object_number=9, tag_tid=1, entry_tid=9861], [object_number=8, tag_tid=1, entry_tid=9860], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=9862' '!=' 'object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_20 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_20 ']' 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-24T01:08:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_20 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_20 2024-04-24T01:08:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:08:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:08:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-24T01:08:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-24T01:08:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=9862], [object_number=9, tag_tid=1, entry_tid=9861], [object_number=8, tag_tid=1, entry_tid=9860], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-24T01:08:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-24T01:08:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=9862], [object_number=9, tag_tid=1, entry_tid=9861], [object_number=8, tag_tid=1, entry_tid=9860], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=9862' '!=' 'object_number=10, tag_tid=1, entry_tid=9862' 2024-04-24T01:08:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:08:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_20 2024-04-24T01:08:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_20 ']' 2024-04-24T01:08:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-24T01:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-24T01:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-24T01:08:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-24T01:08:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_20 2024-04-24T01:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:08:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:08:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:08:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:08:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:08:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:08:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:08:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T01:08:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_20 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:08:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:08:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:08:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:08:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:08:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:08:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:08:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_20 2024-04-24T01:08:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-24T01:08:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-24T01:08:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-24T01:08:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-24T01:08:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:08:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-24T01:08:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-24T01:08:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-24T01:08:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9862], [object_number=11, tag_tid=1, entry_tid=9859]]], state=connected] 2024-04-24T01:08:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.status 2024-04-24T01:08:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-04-24T01:08:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:08:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-04-24T01:08:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_20': 2024-04-24T01:08:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:08:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:08:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122df6cf23d9 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122df6cf23d9 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:00:13 2024 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:00:13 2024 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:01:14 2024 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122df6cf23d9 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:08:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 1fd930ab-4a06-4286-b97b-0518828102db 2024-04-24T01:08:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:08:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_20 snap 2024-04-24T01:08:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:08:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:08:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.export 2024-04-24T01:08:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_20.export 2024-04-24T01:08:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_20.export 2024-04-24T01:08:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.export 2024-04-24T01:08:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_20.export 2024-04-24T01:08:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_20.export 2024-04-24T01:08:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_20.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_20.export 2024-04-24T01:08:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:08:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_21 2024-04-24T01:08:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_21 snap 2024-04-24T01:08:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:08:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-04-24T01:08:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-04-24T01:08:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:08:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-04-24T01:08:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:08:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:08:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:08:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-04-24T01:08:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:08:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:08:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:08:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:08:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:08:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:08:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:08:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-04-24T01:08:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-24T01:08:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:08:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:08:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:08:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-04-24T01:08:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:08:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:08:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:08:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:08:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:08:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:08:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_21 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:08:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:08:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_21 2024-04-24T01:08:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_21 ']' 2024-04-24T01:08:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-24T01:08:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-24T01:08:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-24T01:08:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-24T01:08:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-24T01:08:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-24T01:08:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-24T01:08:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=12585], [object_number=8, tag_tid=1, entry_tid=12584], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-24T01:08:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-24T01:08:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=12585], [object_number=8, tag_tid=1, entry_tid=12584], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=12585' '!=' '' 2024-04-24T01:08:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:08:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=12585' = '' 2024-04-24T01:08:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:08:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:08:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:08:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12585 2024-04-24T01:08:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 12585 2024-04-24T01:08:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_21 2024-04-24T01:08:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_21 ']' 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-24T01:08:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-24T01:08:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-24T01:08:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-24T01:08:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-24T01:08:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=12585], [object_number=8, tag_tid=1, entry_tid=12584], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-24T01:08:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-24T01:08:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=12585], [object_number=8, tag_tid=1, entry_tid=12584], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=12585' '!=' 'object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:08:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_21 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_21 ']' 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-24T01:08:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-24T01:08:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-24T01:08:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-24T01:08:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=12585], [object_number=8, tag_tid=1, entry_tid=12584], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=12585], [object_number=8, tag_tid=1, entry_tid=12584], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=12585' '!=' 'object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_21 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_21 ']' 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-24T01:08:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-24T01:08:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-24T01:08:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-24T01:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-24T01:08:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=12585], [object_number=8, tag_tid=1, entry_tid=12584], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=12585], [object_number=8, tag_tid=1, entry_tid=12584], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=12585' '!=' 'object_number=9, tag_tid=1, entry_tid=12585' 2024-04-24T01:08:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:08:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_21 2024-04-24T01:08:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_21 ']' 2024-04-24T01:08:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-24T01:08:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-24T01:08:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-24T01:08:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-24T01:08:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:08:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:08:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-24T01:08:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:08:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:08:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:08:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:08:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:08:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:08:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:08:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:08:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:08:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:08:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:08:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-24T01:08:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-24T01:08:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-24T01:08:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-24T01:08:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-24T01:08:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12583], [object_number=10, tag_tid=1, entry_tid=12582]]], state=connected] 2024-04-24T01:08:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.status 2024-04-24T01:08:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-04-24T01:08:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:08:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-04-24T01:08:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_21': 2024-04-24T01:08:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:08:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:08:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:08:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122d9f023e65 2024-04-24T01:08:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122d9f023e65 2024-04-24T01:08:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:08:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:08:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:08:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:08:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:00:26 2024 2024-04-24T01:08:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:00:26 2024 2024-04-24T01:08:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:01:27 2024 2024-04-24T01:08:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:08:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122d9f023e65 2024-04-24T01:08:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:08:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:08:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: dae39bca-444c-478c-9b95-c36a9e712976 2024-04-24T01:08:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:08:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_21 snap 2024-04-24T01:08:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:08:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:08:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.export 2024-04-24T01:08:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_21.export 2024-04-24T01:08:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_21.export 2024-04-24T01:08:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.export 2024-04-24T01:08:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_21.export 2024-04-24T01:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:08:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_21.export 2024-04-24T01:08:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_21.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_21.export 2024-04-24T01:08:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:08:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_22 2024-04-24T01:08:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_22 snap 2024-04-24T01:08:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:08:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:08:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-04-24T01:08:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-04-24T01:08:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:08:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:08:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:08:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-04-24T01:08:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:08:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:08:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:08:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-04-24T01:08:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:08:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:08:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:08:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:08:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:08:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:08:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:08:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:08:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:08:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:08:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:08:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-04-24T01:08:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:08:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:08:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:08:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:08:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:08:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:08:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:08:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-04-24T01:08:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:08:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_22 2024-04-24T01:08:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:08:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:08:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:08:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:08:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_22 2024-04-24T01:08:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_22 ']' 2024-04-24T01:08:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-24T01:08:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-24T01:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-24T01:08:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-24T01:08:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-24T01:08:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=8977' '!=' '' 2024-04-24T01:08:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:08:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=8977' = '' 2024-04-24T01:08:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:08:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:08:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:08:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:08:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:08:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:08:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8977 2024-04-24T01:08:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 8977 2024-04-24T01:08:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:08:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:08:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_22 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_22 ']' 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-24T01:08:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-24T01:08:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-24T01:08:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-24T01:08:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-24T01:08:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=8977' '!=' 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:08:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_22 2024-04-24T01:08:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_22 ']' 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-24T01:08:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-24T01:08:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-24T01:08:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=8977' '!=' 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:08:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_22 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_22 ']' 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-24T01:08:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-24T01:08:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-24T01:08:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-24T01:08:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-24T01:08:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-24T01:08:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=8977' '!=' 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:08:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_22 2024-04-24T01:08:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:08:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:08:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:08:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_22 ']' 2024-04-24T01:08:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-24T01:08:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-24T01:08:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:08:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:08:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:08:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:08:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:08:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:08:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:08:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:08:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:08:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-24T01:08:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:08:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:08:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:08:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:08:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:08:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:08:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:08:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-24T01:08:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:08:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-24T01:08:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-24T01:08:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:08:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:08:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:08:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:08:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:08:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:08:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:08:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:08:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:08:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=8977' '!=' 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:08:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:08:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:09:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_22 2024-04-24T01:09:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_22 ']' 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-24T01:09:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-24T01:09:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:09:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-24T01:09:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:09:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:09:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:09:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:09:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:09:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:09:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:09:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:09:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:09:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-24T01:09:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:09:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:09:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:09:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:09:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:09:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8977], [object_number=8, tag_tid=1, entry_tid=8976], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:09:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=8977' '!=' 'object_number=9, tag_tid=1, entry_tid=8977' 2024-04-24T01:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:09:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_22 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_22 ']' 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-24T01:09:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-24T01:09:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-24T01:09:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:09:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-24T01:09:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:09:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:09:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:09:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:09:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:09:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:09:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:09:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:09:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-24T01:09:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-24T01:09:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:09:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:09:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:09:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8975], [object_number=10, tag_tid=1, entry_tid=8974]]], state=connected] 2024-04-24T01:09:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.status 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:09:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-04-24T01:09:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_22': 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123c7ff49db2 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123c7ff49db2 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:00:39 2024 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:00:39 2024 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:01:40 2024 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:09:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123c7ff49db2 2024-04-24T01:09:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:09:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:09:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: c63da6a1-db3e-4796-8aef-3d380b5b78e5 2024-04-24T01:09:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:09:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_22 snap 2024-04-24T01:09:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:09:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:09:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.export 2024-04-24T01:09:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_22.export 2024-04-24T01:09:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_22.export 2024-04-24T01:09:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.export 2024-04-24T01:09:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_22.export 2024-04-24T01:09:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_22.export 2024-04-24T01:09:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_22.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_22.export 2024-04-24T01:09:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:09:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_23 2024-04-24T01:09:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_23 snap 2024-04-24T01:09:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:09:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-04-24T01:09:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-04-24T01:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-04-24T01:09:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:09:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:09:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:09:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-04-24T01:09:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:09:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:09:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:09:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:09:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:09:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:09:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-04-24T01:09:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:09:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:09:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-04-24T01:09:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:09:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:09:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:09:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:09:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:09:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:09:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:09:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-04-24T01:09:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_23 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_23 2024-04-24T01:09:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_23 ']' 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-04-24T01:09:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-04-24T01:09:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3413], [object_number=0, tag_tid=1, entry_tid=3412], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:09:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-04-24T01:09:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3413], [object_number=0, tag_tid=1, entry_tid=3412], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3413' '!=' '' 2024-04-24T01:09:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:09:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3413' = '' 2024-04-24T01:09:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:09:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:09:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:09:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:09:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:09:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:09:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:09:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:09:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:09:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3413 2024-04-24T01:09:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 3413 2024-04-24T01:09:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_23 2024-04-24T01:09:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_23 ']' 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-04-24T01:09:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-04-24T01:09:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-04-24T01:09:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-04-24T01:09:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3413], [object_number=0, tag_tid=1, entry_tid=3412], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:09:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3413], [object_number=0, tag_tid=1, entry_tid=3412], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3413' '!=' 'object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:09:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_23 2024-04-24T01:09:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_23 ']' 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-04-24T01:09:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-04-24T01:09:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-04-24T01:09:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3413], [object_number=0, tag_tid=1, entry_tid=3412], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3413], [object_number=0, tag_tid=1, entry_tid=3412], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3413' '!=' 'object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:09:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_23 2024-04-24T01:09:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_23 ']' 2024-04-24T01:09:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-04-24T01:09:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-04-24T01:09:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-04-24T01:09:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-04-24T01:09:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-04-24T01:09:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-04-24T01:09:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3413], [object_number=0, tag_tid=1, entry_tid=3412], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:09:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-04-24T01:09:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-04-24T01:09:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3413], [object_number=0, tag_tid=1, entry_tid=3412], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3413' '!=' 'object_number=1, tag_tid=1, entry_tid=3413' 2024-04-24T01:09:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:09:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_23 2024-04-24T01:09:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_23 ']' 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-04-24T01:09:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-04-24T01:09:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-04-24T01:09:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:09:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-04-24T01:09:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-04-24T01:09:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:09:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-24T01:09:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-24T01:09:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-24T01:09:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3411], [object_number=2, tag_tid=1, entry_tid=3410]]], state=connected] 2024-04-24T01:09:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.status 2024-04-24T01:09:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:09:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:09:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-04-24T01:09:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:09:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-04-24T01:09:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_23': 2024-04-24T01:09:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:09:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:09:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:09:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123c4b55a8fe 2024-04-24T01:09:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123c4b55a8fe 2024-04-24T01:09:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:09:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:00:50 2024 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:00:50 2024 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:00:50 2024 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123c4b55a8fe 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 92d1f7fd-dd73-4cf1-8fee-d96536f613ba 2024-04-24T01:09:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_23 snap 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.export 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_23.export 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_23.export 2024-04-24T01:09:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.export 2024-04-24T01:09:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_23.export 2024-04-24T01:09:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_23.export 2024-04-24T01:09:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_23.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_23.export 2024-04-24T01:09:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:09:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_24 2024-04-24T01:09:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_24 snap 2024-04-24T01:09:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:09:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:09:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-04-24T01:09:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-04-24T01:09:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:09:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:09:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:09:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:09:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-04-24T01:09:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:09:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:09:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:09:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:09:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:09:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:09:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:09:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:09:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:09:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-04-24T01:09:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:09:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:09:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:09:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:09:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:09:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_24 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:09:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:09:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:09:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_24 2024-04-24T01:09:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:09:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_24 ']' 2024-04-24T01:09:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-04-24T01:09:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-04-24T01:09:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-04-24T01:09:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-04-24T01:09:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_24 2024-04-24T01:09:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-04-24T01:09:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_24 2024-04-24T01:09:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-24T01:09:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-24T01:09:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.status 2024-04-24T01:09:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-24T01:09:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.status 2024-04-24T01:09:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:09:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:09:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11067], [object_number=10, tag_tid=1, entry_tid=11066]]], state=connected] 2024-04-24T01:09:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11067], [object_number=10, tag_tid=1, entry_tid=11066]]], state=connected] 2024-04-24T01:09:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.status 2024-04-24T01:09:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_24 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_24 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-24T01:09:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.status 2024-04-24T01:09:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-24T01:09:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.status 2024-04-24T01:09:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:09:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:09:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11067], [object_number=10, tag_tid=1, entry_tid=11066]]], state=connected] 2024-04-24T01:09:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11067], [object_number=10, tag_tid=1, entry_tid=11066]]], state=connected] 2024-04-24T01:09:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.status 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:09:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-04-24T01:09:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_24': 2024-04-24T01:09:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:09:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:09:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:09:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123c65eda59 2024-04-24T01:09:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123c65eda59 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:00:54 2024 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:00:54 2024 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:01:55 2024 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123c65eda59 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 0e88b4e4-18ac-4fad-8f00-ef91ab47f4dd 2024-04-24T01:09:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:09:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_24 snap 2024-04-24T01:09:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:09:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:09:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.export 2024-04-24T01:09:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_24.export 2024-04-24T01:09:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_24.export 2024-04-24T01:09:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.export 2024-04-24T01:09:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_24.export 2024-04-24T01:09:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_24.export 2024-04-24T01:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_24.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_24.export 2024-04-24T01:09:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:09:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_25 2024-04-24T01:09:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_25 snap 2024-04-24T01:09:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-04-24T01:09:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:09:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:09:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:09:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-04-24T01:09:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:09:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:09:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:09:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:09:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:09:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:09:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:09:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:09:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-04-24T01:09:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:09:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:09:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:09:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:09:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_25 2024-04-24T01:09:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:09:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:09:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_25 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_25 ']' 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-24T01:09:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-24T01:09:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-24T01:09:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-24T01:09:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7713' '!=' '' 2024-04-24T01:09:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:09:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7713' = '' 2024-04-24T01:09:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:09:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:09:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:09:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:09:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:09:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7713 2024-04-24T01:09:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 7713 2024-04-24T01:09:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_25 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_25 ']' 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-24T01:09:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-24T01:09:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7713' '!=' 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:09:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_25 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_25 ']' 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-24T01:09:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-24T01:09:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-24T01:09:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-24T01:09:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-24T01:09:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7713' '!=' 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:09:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_25 2024-04-24T01:09:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_25 ']' 2024-04-24T01:09:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-24T01:09:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-24T01:09:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-24T01:09:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-24T01:09:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-24T01:09:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7713' '!=' 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:09:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_25 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_25 ']' 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-24T01:09:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-24T01:09:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-24T01:09:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-24T01:09:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-24T01:09:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7713' '!=' 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:09:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_25 2024-04-24T01:09:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_25 ']' 2024-04-24T01:09:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-24T01:09:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-24T01:09:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-24T01:09:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-24T01:09:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-24T01:09:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-24T01:09:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-24T01:09:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-24T01:09:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7713], [object_number=4, tag_tid=1, entry_tid=7712], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7713' '!=' 'object_number=5, tag_tid=1, entry_tid=7713' 2024-04-24T01:09:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_25 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_25 ']' 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-24T01:09:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-24T01:09:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-24T01:09:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-24T01:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-24T01:09:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-24T01:09:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7711], [object_number=6, tag_tid=1, entry_tid=7710]]], state=connected] 2024-04-24T01:09:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.status 2024-04-24T01:09:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-04-24T01:09:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:09:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-04-24T01:09:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_25': 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123cd53c4357 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123cd53c4357 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:01:06 2024 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:01:06 2024 2024-04-24T01:09:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:02:07 2024 2024-04-24T01:09:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:09:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123cd53c4357 2024-04-24T01:09:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:09:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:09:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 6508638c-a3d5-415a-b1b7-3f3dbb8ec799 2024-04-24T01:09:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:09:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_25 snap 2024-04-24T01:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.export 2024-04-24T01:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_25.export 2024-04-24T01:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_25.export 2024-04-24T01:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.export 2024-04-24T01:09:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_25.export 2024-04-24T01:09:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_25.export 2024-04-24T01:09:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_25.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_25.export 2024-04-24T01:09:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:09:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_26 2024-04-24T01:09:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_26 snap 2024-04-24T01:09:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:09:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-04-24T01:09:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-04-24T01:09:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:09:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-04-24T01:09:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:09:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:09:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:09:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-04-24T01:09:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:09:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:09:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:09:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:09:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:09:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:09:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-04-24T01:09:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:09:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:09:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:09:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-04-24T01:09:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:09:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:09:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:09:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:09:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:09:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:09:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:09:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-04-24T01:09:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_26 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_26 2024-04-24T01:09:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_26 ']' 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_26' 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2024-04-24T01:09:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2024-04-24T01:09:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_26 2024-04-24T01:09:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:09:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_26 '' 2024-04-24T01:09:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:09:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-04-24T01:09:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_26.status 2024-04-24T01:09:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-04-24T01:09:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:09:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-24T01:09:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:09:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2024-04-24T01:09:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:09:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2024-04-24T01:09:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:09:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-04-24T01:09:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_26.status 2024-04-24T01:09:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-04-24T01:09:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:09:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-24T01:09:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-04-24T01:09:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-04-24T01:09:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=3, tag_tid=2, entry_tid=3' '!=' '' 2024-04-24T01:09:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:09:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=3, tag_tid=2, entry_tid=3' = '' 2024-04-24T01:09:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-04-24T01:09:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:09:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:09:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:09:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-04-24T01:09:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:09:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T01:09:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:09:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:09:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:09:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-04-24T01:09:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:09:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3 2024-04-24T01:09:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 3 2024-04-24T01:09:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_26 2024-04-24T01:09:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_26 ']' 2024-04-24T01:09:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_26' 2024-04-24T01:09:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2024-04-24T01:09:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2024-04-24T01:09:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2024-04-24T01:09:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_26 2024-04-24T01:09:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:09:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_26 '' 2024-04-24T01:09:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:09:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-04-24T01:09:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_26.status 2024-04-24T01:09:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-04-24T01:09:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:09:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:09:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:09:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2024-04-24T01:09:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:09:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2024-04-24T01:09:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:09:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-04-24T01:09:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_26.status 2024-04-24T01:09:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-04-24T01:09:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:09:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:09:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:09:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-24T01:09:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.status 2024-04-24T01:09:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:09:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:09:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-04-24T01:09:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:09:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-04-24T01:09:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_26': 2024-04-24T01:09:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:09:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:09:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:09:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123c5eb25eb4 2024-04-24T01:09:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123c5eb25eb4 2024-04-24T01:09:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:09:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:09:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:09:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:09:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:01:17 2024 2024-04-24T01:09:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:01:17 2024 2024-04-24T01:09:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:02:30 2024 2024-04-24T01:09:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:09:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123c5eb25eb4 2024-04-24T01:09:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:09:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:09:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: f6a308e5-3690-4fda-807d-7defdc598adc 2024-04-24T01:09:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:09:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_26 snap 2024-04-24T01:09:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:09:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:09:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.export 2024-04-24T01:09:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_26.export 2024-04-24T01:09:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_26.export 2024-04-24T01:09:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.export 2024-04-24T01:09:39.430 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:09:39.428+0000 7f6307765640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2024-04-24T01:09:39.430 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:09:39.428+0000 7f6303f5e640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-04-24T01:09:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_26.export 2024-04-24T01:09:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_26.export 2024-04-24T01:09:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_26.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_26.export 2024-04-24T01:09:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:09:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_27 2024-04-24T01:09:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_27 snap 2024-04-24T01:09:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:09:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-04-24T01:09:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-04-24T01:09:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:09:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-04-24T01:09:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:09:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:09:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:09:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-04-24T01:09:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:09:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:09:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:09:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:09:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:09:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:09:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:09:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:09:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:09:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-04-24T01:09:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:09:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:09:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:09:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:09:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:09:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:09:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_27 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:09:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:09:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_27 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_27 ']' 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-24T01:09:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-24T01:09:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-24T01:09:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-24T01:09:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082], [object_number=5, tag_tid=1, entry_tid=6081], [object_number=4, tag_tid=1, entry_tid=6080]]], state=connected] 2024-04-24T01:09:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082], [object_number=5, tag_tid=1, entry_tid=6081], [object_number=4, tag_tid=1, entry_tid=6080]]], state=connected] 2024-04-24T01:09:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=6083' '!=' '' 2024-04-24T01:09:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:09:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=6083' = '' 2024-04-24T01:09:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:09:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:09:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:09:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:09:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:09:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:09:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:09:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:09:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6083 2024-04-24T01:09:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 6083 2024-04-24T01:09:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_27 2024-04-24T01:09:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_27 ']' 2024-04-24T01:09:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-24T01:09:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-24T01:09:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-24T01:09:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-24T01:09:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-24T01:09:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-24T01:09:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082], [object_number=5, tag_tid=1, entry_tid=6081], [object_number=4, tag_tid=1, entry_tid=6080]]], state=connected] 2024-04-24T01:09:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-24T01:09:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-24T01:09:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082], [object_number=5, tag_tid=1, entry_tid=6081], [object_number=4, tag_tid=1, entry_tid=6080]]], state=connected] 2024-04-24T01:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=6083' '!=' 'object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:09:45.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3365857195179104) > weight (0.25) 2024-04-24T01:09:45.453 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T01:09:45.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T01:09:45.453 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T01:09:45.453 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T01:09:45.454 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:09:45.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:09:45.452+0000 7f570fb9e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 96732) UID: 1000 2024-04-24T01:09:45.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:09:45.504+0000 7f5708b90640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x55b1c5ed4700 handle_process_entry_safe: failed to commit journal event: (85) Interrupted system call should be restarted 2024-04-24T01:09:45.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:09:45.504+0000 7f5708b90640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x55b1c5ed4700 handle_replay_complete: replay encountered an error: (85) Interrupted system call should be restarted 2024-04-24T01:09:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_27 2024-04-24T01:09:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_27 ']' 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-24T01:09:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-24T01:09:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-24T01:09:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-24T01:09:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082], [object_number=5, tag_tid=1, entry_tid=6081], [object_number=4, tag_tid=1, entry_tid=6080]]], state=connected] 2024-04-24T01:09:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-24T01:09:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-24T01:09:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082], [object_number=5, tag_tid=1, entry_tid=6081], [object_number=4, tag_tid=1, entry_tid=6080]]], state=connected] 2024-04-24T01:09:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=6083' '!=' 'object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_27 2024-04-24T01:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_27 ']' 2024-04-24T01:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-24T01:09:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-24T01:09:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-24T01:09:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-24T01:09:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-24T01:09:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082], [object_number=5, tag_tid=1, entry_tid=6081], [object_number=4, tag_tid=1, entry_tid=6080]]], state=connected] 2024-04-24T01:09:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082], [object_number=5, tag_tid=1, entry_tid=6081], [object_number=4, tag_tid=1, entry_tid=6080]]], state=connected] 2024-04-24T01:09:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=6083' '!=' 'object_number=7, tag_tid=1, entry_tid=6083' 2024-04-24T01:09:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:09:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_27 2024-04-24T01:09:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_27 ']' 2024-04-24T01:09:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-24T01:09:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-24T01:09:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-24T01:09:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-24T01:09:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-24T01:09:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-24T01:09:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-24T01:09:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:09:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-24T01:09:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-24T01:09:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-24T01:09:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:09:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:09:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6083], [object_number=6, tag_tid=1, entry_tid=6082]]], state=connected] 2024-04-24T01:09:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.status 2024-04-24T01:09:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:09:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-04-24T01:09:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_27': 2024-04-24T01:09:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:09:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:09:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:09:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123cf5909b74 2024-04-24T01:09:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123cf5909b74 2024-04-24T01:09:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:01:21 2024 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:01:21 2024 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:01:21 2024 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123cf5909b74 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: fa1ff108-8ada-4ac8-ac25-88a635c25c42 2024-04-24T01:09:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:09:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_27 snap 2024-04-24T01:09:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:09:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:09:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.export 2024-04-24T01:09:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_27.export 2024-04-24T01:09:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_27.export 2024-04-24T01:09:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.export 2024-04-24T01:09:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_27.export 2024-04-24T01:09:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_27.export 2024-04-24T01:09:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_27.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_27.export 2024-04-24T01:09:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:09:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_28 2024-04-24T01:09:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_28 snap 2024-04-24T01:09:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:09:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:09:51.555 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T01:09:51.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T01:09:51.570 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T01:09:51.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T01:09:51.579 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-04-24T01:09:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-04-24T01:09:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:09:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-04-24T01:09:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:09:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:09:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-04-24T01:09:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:09:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:09:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:09:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:09:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-04-24T01:09:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:09:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:09:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:09:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:09:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:09:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:09:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-04-24T01:09:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:09:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:09:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:09:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-04-24T01:09:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:09:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:09:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:09:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:09:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:09:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_28 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:09:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:09:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:09:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:09:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:09:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:09:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_28 2024-04-24T01:09:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_28 ']' 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:09:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-04-24T01:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:09:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:09:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:09:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:09:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:09:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:09:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-04-24T01:09:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-04-24T01:09:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_28 2024-04-24T01:09:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-04-24T01:09:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_28 2024-04-24T01:09:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:09:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-24T01:09:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-24T01:09:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.status 2024-04-24T01:09:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-24T01:09:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.status 2024-04-24T01:09:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:09:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:09:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13883], [object_number=14, tag_tid=1, entry_tid=13882]]], state=connected] 2024-04-24T01:09:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13883], [object_number=14, tag_tid=1, entry_tid=13882]]], state=connected] 2024-04-24T01:09:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.status 2024-04-24T01:09:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:09:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-04-24T01:09:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_28 2024-04-24T01:09:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-04-24T01:09:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:09:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:09:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_28 2024-04-24T01:09:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:09:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-24T01:09:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-24T01:09:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:09:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.status 2024-04-24T01:09:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-24T01:09:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.status 2024-04-24T01:09:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:09:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:09:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:09:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13883], [object_number=14, tag_tid=1, entry_tid=13882]]], state=connected] 2024-04-24T01:09:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13883], [object_number=14, tag_tid=1, entry_tid=13882]]], state=connected] 2024-04-24T01:09:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.status 2024-04-24T01:09:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:09:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:09:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-04-24T01:09:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:09:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:09:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-04-24T01:09:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_28': 2024-04-24T01:09:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:09:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:09:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:09:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123cc8a9375a 2024-04-24T01:09:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123cc8a9375a 2024-04-24T01:09:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:01:30 2024 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:01:30 2024 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:03:32 2024 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123cc8a9375a 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: d839cf7d-cf05-4344-a46e-941d5468574b 2024-04-24T01:09:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:09:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_28 snap 2024-04-24T01:09:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:09:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:09:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.export 2024-04-24T01:09:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_28.export 2024-04-24T01:09:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_28.export 2024-04-24T01:09:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.export 2024-04-24T01:09:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_28.export 2024-04-24T01:09:54.975 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9492562895574846) > weight (0.25) 2024-04-24T01:09:54.975 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5546301014527749) > weight (0.25) 2024-04-24T01:09:54.975 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.37238513287854813) > weight (0.25) 2024-04-24T01:09:54.975 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5702134215684032) > weight (0.25) 2024-04-24T01:09:54.975 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-04-24T01:09:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:09:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_28.export 2024-04-24T01:09:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_28.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_28.export 2024-04-24T01:09:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:09:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:09:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_29 2024-04-24T01:09:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_29 snap 2024-04-24T01:09:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:09:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:09:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:09:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:09:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-04-24T01:09:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:09:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-04-24T01:09:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:09:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:09:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:09:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:09:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-04-24T01:09:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:09:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:09:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:09:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:09:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:09:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:09:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-04-24T01:09:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:09:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:09:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:09:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:09:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:09:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:09:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:09:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:09:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:09:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:09:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-24T01:09:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:09:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:09:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:09:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:09:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:09:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:09:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:09:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:09:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:09:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:09:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:09:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:09:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:09:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-24T01:09:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:09:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:09:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:09:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-04-24T01:09:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:09:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:09:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:09:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:09:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:09:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:09:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:09:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:09:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:09:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:09:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:09:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:09:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:09:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-04-24T01:09:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:09:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:09:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:09:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-04-24T01:09:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:09:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:09:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:09:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:09:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:09:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:09:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:09:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-04-24T01:09:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:09:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:09:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:09:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 1 2024-04-24T01:09:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:09:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:09:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:09:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:09:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:09:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:09:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:09:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:09:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:09:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:09:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:09:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:09:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:09:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-24T01:09:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:09:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:09:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:09:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-24T01:09:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:09:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:09:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:09:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:09:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:09:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:09:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:09:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:09:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:09:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-24T01:09:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:09:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:09:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:09:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-04-24T01:09:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:09:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:09:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:09:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:09:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:09:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:09:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:09:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:09:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:09:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:09:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:09:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:09:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:09:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:09:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-04-24T01:09:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:09:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:09:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:09:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:09:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-04-24T01:09:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:09:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:09:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:09:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:09:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:09:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:09:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:09:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:09:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:09:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:09:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-04-24T01:09:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:09:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:09:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:09:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:09:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 2 2024-04-24T01:10:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:10:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:10:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-24T01:10:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-24T01:10:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:10:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:10:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-24T01:10:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-04-24T01:10:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:10:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:10:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:10:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-04-24T01:10:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:10:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:10:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:10:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:10:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:10:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:10:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:10:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:10:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:10:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:10:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-04-24T01:10:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 4 2024-04-24T01:10:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:10:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-24T01:10:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:10:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:10:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:10:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-24T01:10:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-24T01:10:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:10:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:10:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-24T01:10:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-04-24T01:10:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:10:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:10:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:10:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-04-24T01:10:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-04-24T01:10:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:10:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:10:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:10:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:10:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:10:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:10:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-04-24T01:10:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 8 2024-04-24T01:10:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:10:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-24T01:10:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-24T01:10:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:10:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:10:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:10:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-24T01:10:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:10:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-04-24T01:10:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-04-24T01:10:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:10:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:10:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:10:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:10:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:10:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:10:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:10:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:10:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:10:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-04-24T01:10:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 8 2024-04-24T01:10:14.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:10:14.964+0000 7f56e85f2640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x556c9bbca000 get_replay_status: replay not running 2024-04-24T01:10:16.033 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:10:16.032+0000 7f6301759640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 9 2024-04-24T01:10:16.086 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:10:16.084+0000 7fbdbe182640 -1 reset not still connected to 0x5622d2201930 2024-04-24T01:10:16.090 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:10:16.088+0000 7fbdbe182640 -1 reset not still connected to 0x5622feea49c0 2024-04-24T01:10:16.090 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:10:16.088+0000 7fbdbe182640 -1 reset not still connected to 0x562309e5e9c0 2024-04-24T01:10:16.110 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:10:16.108+0000 7fbdb0d1c640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-04-24T01:10:16.113 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:10:16.112+0000 7ff576b4d640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 6 2024-04-24T01:10:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:10:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-24T01:10:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:10:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:10:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:10:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-24T01:10:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:10:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:10:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-24T01:10:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:10:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:10:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:10:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:10:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_29 2024-04-24T01:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:10:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:10:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:10:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_29 2024-04-24T01:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_29 ']' 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-04-24T01:10:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T01:10:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2024-04-24T01:10:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2024-04-24T01:10:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-04-24T01:10:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_29 2024-04-24T01:10:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-04-24T01:10:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_29 2024-04-24T01:10:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-24T01:10:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-24T01:10:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.status 2024-04-24T01:10:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-24T01:10:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.status 2024-04-24T01:10:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3255], [object_number=2, tag_tid=1, entry_tid=3254]]], state=connected] 2024-04-24T01:10:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3255], [object_number=2, tag_tid=1, entry_tid=3254]]], state=connected] 2024-04-24T01:10:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.status 2024-04-24T01:10:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-04-24T01:10:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_29 2024-04-24T01:10:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-04-24T01:10:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_29 2024-04-24T01:10:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-24T01:10:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-24T01:10:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.status 2024-04-24T01:10:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-24T01:10:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.status 2024-04-24T01:10:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3255], [object_number=2, tag_tid=1, entry_tid=3254]]], state=connected] 2024-04-24T01:10:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3255], [object_number=2, tag_tid=1, entry_tid=3254]]], state=connected] 2024-04-24T01:10:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.status 2024-04-24T01:10:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-04-24T01:10:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:10:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:10:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:10:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:10:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-04-24T01:10:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_29': 2024-04-24T01:10:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:10:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:10:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:10:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b246367a49 2024-04-24T01:10:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b246367a49 2024-04-24T01:10:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:10:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:10:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:01:42 2024 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:01:42 2024 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:02:43 2024 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10b246367a49 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 2c1332ec-b4ea-4a0b-8e7c-ca8f5160a038 2024-04-24T01:10:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:10:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_29 snap 2024-04-24T01:10:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:10:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:10:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.export 2024-04-24T01:10:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_29.export 2024-04-24T01:10:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_29.export 2024-04-24T01:10:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.export 2024-04-24T01:10:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:10:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_29.export 2024-04-24T01:10:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:10:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_29.export 2024-04-24T01:10:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_29.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_29.export 2024-04-24T01:10:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:10:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_30 2024-04-24T01:10:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_30 snap 2024-04-24T01:10:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:10:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-04-24T01:10:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-04-24T01:10:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:10:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:10:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:10:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-04-24T01:10:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:10:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:10:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:10:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-04-24T01:10:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:10:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:10:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:10:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:10:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:10:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:10:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-04-24T01:10:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:10:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:10:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:10:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:10:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-04-24T01:10:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:10:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:10:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-04-24T01:10:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-04-24T01:10:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:10:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:10:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:10:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-04-24T01:10:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:10:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:10:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:10:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:10:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:10:27.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_30 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:10:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:10:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_30 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_30 ']' 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-24T01:10:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_30 2024-04-24T01:10:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:10:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:10:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:10:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-24T01:10:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-24T01:10:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-24T01:10:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-24T01:10:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-24T01:10:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=2898], [object_number=1, tag_tid=1, entry_tid=2897], [object_number=0, tag_tid=1, entry_tid=2896], [object_number=3, tag_tid=1, entry_tid=2895]]], state=connected] 2024-04-24T01:10:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-24T01:10:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-24T01:10:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-24T01:10:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=2898], [object_number=1, tag_tid=1, entry_tid=2897], [object_number=0, tag_tid=1, entry_tid=2896], [object_number=3, tag_tid=1, entry_tid=2895]]], state=connected] 2024-04-24T01:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=2898' '!=' '' 2024-04-24T01:10:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:10:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=2898' = '' 2024-04-24T01:10:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:10:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:10:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:10:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=2898 2024-04-24T01:10:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 1 -a 1 -le 2898 2024-04-24T01:10:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_30 2024-04-24T01:10:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_30 ']' 2024-04-24T01:10:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-24T01:10:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-24T01:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-24T01:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_30 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:10:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:10:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-24T01:10:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-24T01:10:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-24T01:10:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-24T01:10:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-24T01:10:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=2898], [object_number=1, tag_tid=1, entry_tid=2897], [object_number=0, tag_tid=1, entry_tid=2896], [object_number=3, tag_tid=1, entry_tid=2895]]], state=connected] 2024-04-24T01:10:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-24T01:10:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-24T01:10:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-24T01:10:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-24T01:10:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-24T01:10:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=2898], [object_number=1, tag_tid=1, entry_tid=2897], [object_number=0, tag_tid=1, entry_tid=2896], [object_number=3, tag_tid=1, entry_tid=2895]]], state=connected] 2024-04-24T01:10:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=2898' '!=' 'object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:10:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_30 2024-04-24T01:10:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_30 ']' 2024-04-24T01:10:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-24T01:10:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-24T01:10:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-24T01:10:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_30 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:10:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:10:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-24T01:10:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-24T01:10:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-24T01:10:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-24T01:10:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-24T01:10:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=3062], [object_number=1, tag_tid=1, entry_tid=3061], [object_number=0, tag_tid=1, entry_tid=3060], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-24T01:10:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-24T01:10:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-24T01:10:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-24T01:10:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-24T01:10:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=3062], [object_number=1, tag_tid=1, entry_tid=3061], [object_number=0, tag_tid=1, entry_tid=3060], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=2, tag_tid=1, entry_tid=3062' 2024-04-24T01:10:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=2, tag_tid=1, entry_tid=3062' 2024-04-24T01:10:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=3062' '!=' 'object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:10:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=3062' = 'object_number=2, tag_tid=1, entry_tid=2898' 2024-04-24T01:10:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=2, tag_tid=1, entry_tid=3062' 2024-04-24T01:10:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:10:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=2, tag_tid=1, entry_tid=3062' 2024-04-24T01:10:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:10:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:10:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=2, tag_tid=1, entry_tid=3062' 2024-04-24T01:10:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3062 2024-04-24T01:10:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 1 -a 1 -le 3062 2024-04-24T01:10:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_30 2024-04-24T01:10:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_30 ']' 2024-04-24T01:10:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-24T01:10:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-24T01:10:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:30.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-24T01:10:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-24T01:10:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T01:10:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_30 2024-04-24T01:10:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:10:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:10:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:10:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_30 2024-04-24T01:10:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-24T01:10:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-24T01:10:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-24T01:10:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-24T01:10:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-24T01:10:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-24T01:10:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-24T01:10:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:10:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-24T01:10:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-24T01:10:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-24T01:10:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3062], [object_number=3, tag_tid=1, entry_tid=3059]]], state=connected] 2024-04-24T01:10:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.status 2024-04-24T01:10:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:10:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-04-24T01:10:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:10:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:10:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:10:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-04-24T01:10:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_30': 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d83bd99a4d 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d83bd99a4d 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:01:48 2024 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:01:48 2024 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:03:54 2024 2024-04-24T01:10:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:10:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d83bd99a4d 2024-04-24T01:10:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:10:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:10:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: bf7d5dd9-3a26-484b-b4a3-1b53efd4662a 2024-04-24T01:10:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:10:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_30 snap 2024-04-24T01:10:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:10:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:10:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.export 2024-04-24T01:10:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_30.export 2024-04-24T01:10:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_30.export 2024-04-24T01:10:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.export 2024-04-24T01:10:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:10:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_30.export 2024-04-24T01:10:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:10:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_30.export 2024-04-24T01:10:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_30.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_30.export 2024-04-24T01:10:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:10:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_31 2024-04-24T01:10:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_31 snap 2024-04-24T01:10:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:10:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-04-24T01:10:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-04-24T01:10:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:10:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:10:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:10:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-04-24T01:10:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:10:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:10:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-04-24T01:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:10:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:10:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:10:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:10:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-04-24T01:10:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T01:10:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-04-24T01:10:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:10:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:10:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:10:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-04-24T01:10:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:10:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:10:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:10:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:10:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:10:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:10:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:10:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-04-24T01:10:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_31 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_31 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_31 ']' 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-24T01:10:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T01:10:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-24T01:10:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-24T01:10:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-24T01:10:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-24T01:10:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-24T01:10:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10649' '!=' '' 2024-04-24T01:10:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:10:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10649' = '' 2024-04-24T01:10:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:10:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:10:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:10:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10649 2024-04-24T01:10:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 10649 2024-04-24T01:10:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_31 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_31 ']' 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-24T01:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T01:10:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-24T01:10:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-24T01:10:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10649' '!=' 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_31 2024-04-24T01:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_31 ']' 2024-04-24T01:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-24T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-24T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T01:10:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-24T01:10:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-24T01:10:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-24T01:10:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-24T01:10:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-24T01:10:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10649' '!=' 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:10:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_31 2024-04-24T01:10:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_31 ']' 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-24T01:10:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T01:10:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-24T01:10:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-24T01:10:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-24T01:10:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-24T01:10:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-24T01:10:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10649' '!=' 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:10:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_31 2024-04-24T01:10:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_31 ']' 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-24T01:10:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T01:10:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-24T01:10:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-24T01:10:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-24T01:10:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-24T01:10:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-24T01:10:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10649], [object_number=8, tag_tid=1, entry_tid=10648], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10649' '!=' 'object_number=9, tag_tid=1, entry_tid=10649' 2024-04-24T01:10:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:10:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_31 2024-04-24T01:10:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_31 ']' 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-24T01:10:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T01:10:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-24T01:10:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-24T01:10:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-24T01:10:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-24T01:10:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-24T01:10:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10647], [object_number=10, tag_tid=1, entry_tid=10646]]], state=connected] 2024-04-24T01:10:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.status 2024-04-24T01:10:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-04-24T01:10:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:10:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:10:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:10:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-04-24T01:10:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_31': 2024-04-24T01:10:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:10:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:10:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:10:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122d4f1beb1 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122d4f1beb1 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:01:53 2024 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:01:53 2024 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:02:54 2024 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122d4f1beb1 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 887e756f-591b-40ab-86a2-1db5d474c150 2024-04-24T01:10:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:10:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_31 snap 2024-04-24T01:10:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:10:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:10:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.export 2024-04-24T01:10:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_31.export 2024-04-24T01:10:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_31.export 2024-04-24T01:10:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.export 2024-04-24T01:10:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:10:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_31.export 2024-04-24T01:10:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:10:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_31.export 2024-04-24T01:10:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_31.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_31.export 2024-04-24T01:10:45.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:10:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_32 2024-04-24T01:10:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_32 snap 2024-04-24T01:10:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:10:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:10:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-04-24T01:10:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:10:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:10:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:10:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-04-24T01:10:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:10:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:10:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:10:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:10:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:10:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:10:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:10:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-04-24T01:10:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:10:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:10:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:10:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:10:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_32 2024-04-24T01:10:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:10:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:10:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_32 2024-04-24T01:10:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_32 ']' 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-24T01:10:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-24T01:10:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-24T01:10:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-24T01:10:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=4018], [object_number=1, tag_tid=1, entry_tid=4017], [object_number=0, tag_tid=1, entry_tid=4016], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:10:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-24T01:10:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-24T01:10:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=4018], [object_number=1, tag_tid=1, entry_tid=4017], [object_number=0, tag_tid=1, entry_tid=4016], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=4018' '!=' '' 2024-04-24T01:10:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:10:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=4018' = '' 2024-04-24T01:10:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:10:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:10:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:10:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:10:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:10:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=4018 2024-04-24T01:10:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 4018 2024-04-24T01:10:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_32 2024-04-24T01:10:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_32 ']' 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-24T01:10:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-24T01:10:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-24T01:10:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-24T01:10:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=4018], [object_number=1, tag_tid=1, entry_tid=4017], [object_number=0, tag_tid=1, entry_tid=4016], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:10:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-24T01:10:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-24T01:10:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=4018], [object_number=1, tag_tid=1, entry_tid=4017], [object_number=0, tag_tid=1, entry_tid=4016], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=4018' '!=' 'object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_32 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_32 ']' 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-24T01:10:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-24T01:10:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-24T01:10:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=4018], [object_number=1, tag_tid=1, entry_tid=4017], [object_number=0, tag_tid=1, entry_tid=4016], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:10:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-24T01:10:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-24T01:10:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=4018], [object_number=1, tag_tid=1, entry_tid=4017], [object_number=0, tag_tid=1, entry_tid=4016], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=4018' '!=' 'object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:10:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_32 2024-04-24T01:10:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_32 ']' 2024-04-24T01:10:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-24T01:10:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-24T01:10:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-24T01:10:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-24T01:10:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-24T01:10:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-24T01:10:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=4018], [object_number=1, tag_tid=1, entry_tid=4017], [object_number=0, tag_tid=1, entry_tid=4016], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=4018], [object_number=1, tag_tid=1, entry_tid=4017], [object_number=0, tag_tid=1, entry_tid=4016], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=4018' '!=' 'object_number=2, tag_tid=1, entry_tid=4018' 2024-04-24T01:10:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:10:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_32 2024-04-24T01:10:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_32 ']' 2024-04-24T01:10:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-24T01:10:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-24T01:10:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-24T01:10:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-24T01:10:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-24T01:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-24T01:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:10:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-24T01:10:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-24T01:10:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:10:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-24T01:10:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-24T01:10:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-24T01:10:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:10:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:10:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=4018], [object_number=3, tag_tid=1, entry_tid=4015]]], state=connected] 2024-04-24T01:10:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.status 2024-04-24T01:10:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:10:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:10:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-04-24T01:10:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:10:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:10:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:10:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-04-24T01:10:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_32': 2024-04-24T01:10:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:10:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:10:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:10:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123ccfd45984 2024-04-24T01:10:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123ccfd45984 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:02:03 2024 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:02:03 2024 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:02:03 2024 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123ccfd45984 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: a027927c-19cf-4d76-95e7-2029e6e07b6c 2024-04-24T01:10:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_32 snap 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.export 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_32.export 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_32.export 2024-04-24T01:10:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.export 2024-04-24T01:10:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:10:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_32.export 2024-04-24T01:10:55.037 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:10:55.036+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:10:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:10:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_32.export 2024-04-24T01:10:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_32.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_32.export 2024-04-24T01:10:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:10:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_33 2024-04-24T01:10:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_33 snap 2024-04-24T01:10:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:10:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:10:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:10:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-04-24T01:10:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:10:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:10:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:10:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:10:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-04-24T01:10:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:10:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:10:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:10:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:10:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:10:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:10:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-24T01:10:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:10:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-24T01:10:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:10:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-24T01:10:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:10:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:10:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:10:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-24T01:10:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:10:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:10:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:10:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-04-24T01:10:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:10:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:10:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:10:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:10:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:10:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:10:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:10:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-04-24T01:10:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:10:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:10:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:10:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:10:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:10:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:10:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:10:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-04-24T01:10:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_33 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_33 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_33 ']' 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:10:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:10:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:10:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:10:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:10:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:10:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:10:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:10:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:10:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:10:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:10:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:10:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:10:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:10:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:10:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:10:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:10:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:10:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:10:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:10:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-24T01:10:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:10:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-24T01:10:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:10:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:10:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:10:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:10:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:10:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:10:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:10:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:10:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:10:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:10:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:10:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:10:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:10:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:10:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:10:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:10:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:10:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:10:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:10:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:10:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:10:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:10:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:10:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10985' '!=' '' 2024-04-24T01:10:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:10:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10985' = '' 2024-04-24T01:10:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:10:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:10:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:10:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:10:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:10:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:10:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:10:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:10:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:10:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:10:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10985 2024-04-24T01:10:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 10985 2024-04-24T01:10:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:10:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:10:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:10:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_33 2024-04-24T01:10:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:10:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_33 ']' 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:10:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:10:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:10:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:10:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:10:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:10:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:10:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:10:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:10:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:10:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:10:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:10:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:10:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:10:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:10:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:10:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:10:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:10:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:10:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:10:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:10:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:10:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:10:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:10:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:10:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:10:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:10:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:10:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:10:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:10:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:10:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:10:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:10:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:10:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-24T01:11:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-24T01:11:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-24T01:11:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-24T01:11:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10985' '!=' 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:11:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_33 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_33 ']' 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-24T01:11:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-24T01:11:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10985' '!=' 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:11:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_33 2024-04-24T01:11:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:11:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_33 ']' 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10985' '!=' 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_33 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_33 ']' 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-24T01:11:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-24T01:11:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-24T01:11:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10985], [object_number=8, tag_tid=1, entry_tid=10984], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10985' '!=' 'object_number=9, tag_tid=1, entry_tid=10985' 2024-04-24T01:11:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:11:06.580 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9065019282957448) > weight (0.25) 2024-04-24T01:11:06.581 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4001282911340909) > weight (0.25) 2024-04-24T01:11:06.581 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5703024245788686) > weight (0.25) 2024-04-24T01:11:06.581 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-24T01:11:06.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-24T01:11:06.581 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-24T01:11:06.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:11:06.578+0000 7f2231a3b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 102367) UID: 1000 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_33 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_33 ']' 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T01:11:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:11:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T01:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:11:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T01:11:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 4 2024-04-24T01:11:15.582 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-24T01:11:15.583 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:11:15.583 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-24T01:11:15.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-24T01:11:15.583 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-24T01:11:15.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-24T01:11:15.586 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-04-24T01:11:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T01:11:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T01:11:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T01:11:34.849 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:11:34.845+0000 7f63163ce640 -1 reset not still connected to 0x55c563dc6750 2024-04-24T01:11:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-24T01:11:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-24T01:11:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_33 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T01:11:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_33 2024-04-24T01:11:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_33 2024-04-24T01:11:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-24T01:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-24T01:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:11:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-24T01:11:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-24T01:11:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-24T01:11:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10983], [object_number=10, tag_tid=1, entry_tid=10982]]], state=connected] 2024-04-24T01:11:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.status 2024-04-24T01:11:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-04-24T01:11:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:11:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:11:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:11:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:11:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:11:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-04-24T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_33': 2024-04-24T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b2987a7bb9 2024-04-24T01:11:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b2987a7bb9 2024-04-24T01:11:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:11:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:11:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:11:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:11:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:02:09 2024 2024-04-24T01:11:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:02:09 2024 2024-04-24T01:11:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:04:26 2024 2024-04-24T01:11:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:11:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10b2987a7bb9 2024-04-24T01:11:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:11:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:11:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 3b5b62ba-0f91-4e0d-a65e-deab6e5e850b 2024-04-24T01:11:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:11:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_33 snap 2024-04-24T01:11:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:11:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:11:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:11:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.export 2024-04-24T01:11:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_33.export 2024-04-24T01:11:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_33.export 2024-04-24T01:11:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.export 2024-04-24T01:11:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:11:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_33.export 2024-04-24T01:11:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:11:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_33.export 2024-04-24T01:11:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_33.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_33.export 2024-04-24T01:11:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:11:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_34 2024-04-24T01:11:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_34 snap 2024-04-24T01:11:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:11:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:11:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:11:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-04-24T01:11:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-04-24T01:11:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:11:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:11:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:11:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-04-24T01:11:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:11:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:11:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:11:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-04-24T01:11:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:11:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:11:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:11:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:11:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:11:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:11:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:11:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-04-24T01:11:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:11:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:11:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:11:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:11:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:11:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-04-24T01:11:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:11:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-04-24T01:11:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:11:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:11:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:11:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:11:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:11:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:11:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-04-24T01:11:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:11:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2024-04-24T01:11:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:11:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:11:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:11:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:11:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2024-04-24T01:11:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:11:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:11:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:11:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:11:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:11:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:11:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:11:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:11:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:11:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:11:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:11:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:11:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2024-04-24T01:11:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:11:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:11:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:11:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:11:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:11:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_34 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:11:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:11:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:11:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:11:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_34 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_34 ']' 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-24T01:11:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-24T01:11:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_34 2024-04-24T01:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-24T01:11:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-24T01:11:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-24T01:11:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:11:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-24T01:11:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:11:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:11:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:11:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:11:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:11:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:11:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T01:11:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:11:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:11:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:11:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:11:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:11:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:11:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=1 2024-04-24T01:11:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-24T01:11:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:11:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:11:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_34 2024-04-24T01:11:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_34 ']' 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-24T01:11:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-24T01:11:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_34 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-24T01:11:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-24T01:11:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:11:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-24T01:11:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:11:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_34 2024-04-24T01:11:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_34 ']' 2024-04-24T01:11:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-24T01:11:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-24T01:11:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-24T01:11:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2024-04-24T01:11:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_34 2024-04-24T01:11:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-24T01:11:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-24T01:11:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:11:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-24T01:11:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-24T01:11:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_34 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_34 ']' 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-24T01:11:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_34 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-24T01:11:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-24T01:11:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:11:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:11:51.976 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7880119798856952) > weight (0.25) 2024-04-24T01:11:51.976 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T01:11:51.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T01:11:51.977 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-24T01:11:51.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:11:51.977+0000 7f76a4a5d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 105513) UID: 1000 2024-04-24T01:11:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_34 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_34 ']' 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-24T01:11:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_34 2024-04-24T01:11:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:11:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:11:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:11:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:11:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T01:11:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_34 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:11:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:11:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:11:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:11:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:11:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:11:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:11:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_34 2024-04-24T01:11:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-24T01:11:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-24T01:11:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:11:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-24T01:11:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-24T01:11:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:11:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-24T01:11:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-24T01:11:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-24T01:11:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:11:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:11:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4111], [object_number=6, tag_tid=1, entry_tid=4110]]], state=connected] 2024-04-24T01:11:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.status 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:11:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:11:53.978 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T01:11:53.978 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:11:53.978 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T01:11:53.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T01:11:53.979 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T01:11:53.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T01:11:53.982 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-04-24T01:11:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_34': 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d89132d290 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d89132d290 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:11:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:02:24 2024 2024-04-24T01:11:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:02:24 2024 2024-04-24T01:11:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:03:25 2024 2024-04-24T01:11:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:11:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d89132d290 2024-04-24T01:11:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:11:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:11:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 899dbee7-a87c-49e6-8f81-ef5e4b69ac58 2024-04-24T01:11:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:11:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_34 snap 2024-04-24T01:11:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:11:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:11:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:11:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.export 2024-04-24T01:11:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_34.export 2024-04-24T01:11:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_34.export 2024-04-24T01:11:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.export 2024-04-24T01:11:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:11:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_34.export 2024-04-24T01:11:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:11:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_34.export 2024-04-24T01:11:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_34.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_34.export 2024-04-24T01:11:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:11:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_35 2024-04-24T01:11:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_35 snap 2024-04-24T01:11:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:11:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:11:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:11:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:11:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:11:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:11:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-04-24T01:11:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:11:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:11:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:11:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:11:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-04-24T01:11:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:11:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:11:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:11:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:11:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:11:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:11:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:11:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-04-24T01:11:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:11:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:11:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:11:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-04-24T01:11:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T01:11:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:11:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:11:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:11:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:11:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:11:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:11:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-04-24T01:11:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:11:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:11:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:11:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:11:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:11:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:11:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:11:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-04-24T01:11:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:11:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_35 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:11:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_35 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_35 ']' 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-24T01:11:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-24T01:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T01:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-24T01:11:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-24T01:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-24T01:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:11:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:11:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:11:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-24T01:11:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-24T01:11:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:11:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:11:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:11:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9025' '!=' '' 2024-04-24T01:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9025' = '' 2024-04-24T01:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:11:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:11:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:11:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:11:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:11:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:11:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:11:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:11:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:11:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9025 2024-04-24T01:11:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 9025 2024-04-24T01:11:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:11:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_35 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_35 ']' 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-24T01:11:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-24T01:11:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T01:11:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-24T01:11:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-24T01:11:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:11:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:11:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:11:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-24T01:11:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-24T01:11:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:11:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:11:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:11:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9025' '!=' 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_35 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_35 ']' 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:11:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:11:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:11:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:11:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:11:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:11:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:11:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:11:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:11:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:11:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:11:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:11:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:11:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:11:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:11:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:11:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:11:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-24T01:11:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T01:11:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:11:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:11:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:11:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-24T01:11:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:11:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:11:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:11:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:11:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:11:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:11:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:11:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:11:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:11:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:11:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-24T01:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-24T01:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-24T01:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:11:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:11:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:11:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:11:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:11:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-24T01:11:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-24T01:11:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:11:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:11:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:11:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:11:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:11:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:11:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:11:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:11:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:11:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:11:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:11:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:11:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:11:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9025' '!=' 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:11:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:11:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_35 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_35 ']' 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-24T01:12:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-24T01:12:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T01:12:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-24T01:12:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-24T01:12:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-24T01:12:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-24T01:12:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:12:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:12:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:12:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-24T01:12:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-24T01:12:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:12:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:12:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:12:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:12:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:12:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9025' '!=' 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:12:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:12:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_35 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_35 ']' 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-24T01:12:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-24T01:12:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T01:12:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-24T01:12:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-24T01:12:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-24T01:12:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:12:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:12:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:12:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:12:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:12:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9025], [object_number=8, tag_tid=1, entry_tid=9024], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:12:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:12:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9025' '!=' 'object_number=9, tag_tid=1, entry_tid=9025' 2024-04-24T01:12:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_35 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_35 ']' 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-24T01:12:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T01:12:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-24T01:12:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-24T01:12:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-24T01:12:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-24T01:12:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:12:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:12:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:12:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-24T01:12:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-24T01:12:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9023], [object_number=10, tag_tid=1, entry_tid=9022]]], state=connected] 2024-04-24T01:12:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.status 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:12:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-04-24T01:12:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_35': 2024-04-24T01:12:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:12:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122dfb3da8a7 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122dfb3da8a7 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:02:28 2024 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:02:28 2024 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:03:29 2024 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122dfb3da8a7 2024-04-24T01:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:12:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:12:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 0bef3c84-25b8-469f-9fc9-8ff0d4019a6e 2024-04-24T01:12:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:12:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_35 snap 2024-04-24T01:12:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:12:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:12:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.export 2024-04-24T01:12:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_35.export 2024-04-24T01:12:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_35.export 2024-04-24T01:12:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.export 2024-04-24T01:12:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_35.export 2024-04-24T01:12:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_35.export 2024-04-24T01:12:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_35.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_35.export 2024-04-24T01:12:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:12:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_36 2024-04-24T01:12:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_36 snap 2024-04-24T01:12:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:12:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-04-24T01:12:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:12:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:12:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:12:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-04-24T01:12:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:12:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:12:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:12:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:12:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:12:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:12:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-04-24T01:12:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:12:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:12:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:12:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:12:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:12:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-04-24T01:12:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:12:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:12:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:12:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:12:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:12:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:12:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:12:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-04-24T01:12:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:12:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_36 2024-04-24T01:12:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:12:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:12:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_36 2024-04-24T01:12:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_36 ']' 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-24T01:12:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-24T01:12:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-24T01:12:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-24T01:12:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-24T01:12:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-24T01:12:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-24T01:12:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:12:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:12:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=5760], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757]]], state=connected] 2024-04-24T01:12:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:10.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-24T01:12:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-24T01:12:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-24T01:12:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:12:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:12:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=5760], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757]]], state=connected] 2024-04-24T01:12:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=5760' '!=' '' 2024-04-24T01:12:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:12:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=5760' = '' 2024-04-24T01:12:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:12:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:12:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:12:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:12:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:12:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=5760 2024-04-24T01:12:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 5760 2024-04-24T01:12:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_36 2024-04-24T01:12:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_36 ']' 2024-04-24T01:12:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-24T01:12:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-24T01:12:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-24T01:12:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-24T01:12:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-24T01:12:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-24T01:12:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-24T01:12:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-24T01:12:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-24T01:12:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:12:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:12:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=5760], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757]]], state=connected] 2024-04-24T01:12:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:12:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-24T01:12:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-24T01:12:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-24T01:12:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-24T01:12:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-24T01:12:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:12:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:12:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=5760], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757]]], state=connected] 2024-04-24T01:12:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=5760' '!=' 'object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:12:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_36 2024-04-24T01:12:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_36 ']' 2024-04-24T01:12:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-24T01:12:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-24T01:12:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-24T01:12:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-24T01:12:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:11.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-24T01:12:11.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:11.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:11.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:11.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-24T01:12:11.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-24T01:12:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-24T01:12:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-24T01:12:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:12:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:12:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=5760], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757]]], state=connected] 2024-04-24T01:12:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:12:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-24T01:12:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-24T01:12:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-24T01:12:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-24T01:12:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-24T01:12:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:12:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:12:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=5760], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757]]], state=connected] 2024-04-24T01:12:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=5760' '!=' 'object_number=4, tag_tid=1, entry_tid=5760' 2024-04-24T01:12:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:12:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_36 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_36 ']' 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-24T01:12:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-24T01:12:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-24T01:12:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-24T01:12:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-24T01:12:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-24T01:12:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-24T01:12:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:12:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:12:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-24T01:12:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-24T01:12:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:12:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:12:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-24T01:12:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.status 2024-04-24T01:12:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:12:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:12:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-04-24T01:12:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:12:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-04-24T01:12:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_36': 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123cb6b15adf 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123cb6b15adf 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:02:40 2024 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:02:40 2024 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:02:40 2024 2024-04-24T01:12:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:12:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123cb6b15adf 2024-04-24T01:12:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:12:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:12:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: d9ef14ae-65bb-4ce7-b745-30b1926864dd 2024-04-24T01:12:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:12:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_36 snap 2024-04-24T01:12:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:12:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:12:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.export 2024-04-24T01:12:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_36.export 2024-04-24T01:12:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_36.export 2024-04-24T01:12:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.export 2024-04-24T01:12:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_36.export 2024-04-24T01:12:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_36.export 2024-04-24T01:12:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_36.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_36.export 2024-04-24T01:12:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:12:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_37 2024-04-24T01:12:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_37 snap 2024-04-24T01:12:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:12:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-04-24T01:12:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:12:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:12:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:12:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-04-24T01:12:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:12:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:12:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:12:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:12:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:12:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:12:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-04-24T01:12:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:12:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:12:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:12:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-04-24T01:12:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:12:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-04-24T01:12:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:12:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:12:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:12:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-04-24T01:12:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:12:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-04-24T01:12:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:12:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:12:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:12:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-04-24T01:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_37 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:12:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:12:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:12:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:12:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:12:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:12:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:12:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_37 2024-04-24T01:12:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:12:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:12:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:12:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:12:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:12:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_37 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:12:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:12:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_37 2024-04-24T01:12:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_37 ']' 2024-04-24T01:12:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-04-24T01:12:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-04-24T01:12:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-04-24T01:12:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2024-04-24T01:12:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_37 2024-04-24T01:12:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:12:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_37 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:12:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:12:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:12:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:12:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-04-24T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-04-24T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-24T01:12:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-24T01:12:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-24T01:12:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250], [object_number=9, tag_tid=1, entry_tid=10249], [object_number=8, tag_tid=1, entry_tid=10248]]], state=connected] 2024-04-24T01:12:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-04-24T01:12:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-04-24T01:12:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-24T01:12:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-24T01:12:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-24T01:12:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250], [object_number=9, tag_tid=1, entry_tid=10249], [object_number=8, tag_tid=1, entry_tid=10248]]], state=connected] 2024-04-24T01:12:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=10251' '!=' '' 2024-04-24T01:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=10251' = '' 2024-04-24T01:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:12:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:12:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:12:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10251 2024-04-24T01:12:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 10251 2024-04-24T01:12:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_37 2024-04-24T01:12:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_37 ']' 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-04-24T01:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2024-04-24T01:12:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_37 2024-04-24T01:12:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:12:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:12:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_37 2024-04-24T01:12:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:12:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:12:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:12:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:12:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:12:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:12:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:12:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:12:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:12:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-04-24T01:12:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-04-24T01:12:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-24T01:12:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-24T01:12:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-24T01:12:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250], [object_number=9, tag_tid=1, entry_tid=10249], [object_number=8, tag_tid=1, entry_tid=10248]]], state=connected] 2024-04-24T01:12:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-24T01:12:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-24T01:12:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-24T01:12:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250], [object_number=9, tag_tid=1, entry_tid=10249], [object_number=8, tag_tid=1, entry_tid=10248]]], state=connected] 2024-04-24T01:12:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=10251' '!=' 'object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:12:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_37 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_37 ']' 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_37 2024-04-24T01:12:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:12:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:12:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_37 2024-04-24T01:12:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:12:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:12:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:12:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:12:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:12:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:12:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:12:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:12:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:12:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-04-24T01:12:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-04-24T01:12:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-24T01:12:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-24T01:12:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-24T01:12:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250], [object_number=9, tag_tid=1, entry_tid=10249], [object_number=8, tag_tid=1, entry_tid=10248]]], state=connected] 2024-04-24T01:12:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-24T01:12:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-24T01:12:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-24T01:12:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250], [object_number=9, tag_tid=1, entry_tid=10249], [object_number=8, tag_tid=1, entry_tid=10248]]], state=connected] 2024-04-24T01:12:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=10251' '!=' 'object_number=11, tag_tid=1, entry_tid=10251' 2024-04-24T01:12:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_37 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_37 ']' 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-04-24T01:12:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2024-04-24T01:12:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_37 2024-04-24T01:12:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:12:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:12:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:12:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:12:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T01:12:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_37 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:12:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:12:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:12:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:12:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:12:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:12:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_37 2024-04-24T01:12:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-04-24T01:12:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-04-24T01:12:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-24T01:12:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-24T01:12:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-24T01:12:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-24T01:12:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-24T01:12:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10251], [object_number=10, tag_tid=1, entry_tid=10250]]], state=connected] 2024-04-24T01:12:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.status 2024-04-24T01:12:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-04-24T01:12:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:12:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:23.336 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:12:23.336+0000 7fbdbe182640 -1 reset not still connected to 0x5622dab380d0 2024-04-24T01:12:23.336 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:12:23.336+0000 7fbdbe182640 -1 reset not still connected to 0x5622dab38b60 2024-04-24T01:12:23.336 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:12:23.336+0000 7fbdbe182640 -1 reset not still connected to 0x5622e21e1ee0 2024-04-24T01:12:23.336 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:12:23.336+0000 7fbdbe182640 -1 reset not still connected to 0x5622f04c5450 2024-04-24T01:12:23.336 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:12:23.336+0000 7fbdbe182640 -1 reset not still connected to 0x5622f4b9f5f0 2024-04-24T01:12:23.336 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:12:23.336+0000 7fbdbe182640 -1 reset not still connected to 0x5622f9f565b0 2024-04-24T01:12:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_37': 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b241e7e5cc 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b241e7e5cc 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:12:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:02:48 2024 2024-04-24T01:12:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:02:48 2024 2024-04-24T01:12:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:04:26 2024 2024-04-24T01:12:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:12:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10b241e7e5cc 2024-04-24T01:12:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:12:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:12:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 58a9a543-2718-4d39-8b8d-21ef40a01ed2 2024-04-24T01:12:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:12:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_37 snap 2024-04-24T01:12:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:12:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:12:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.export 2024-04-24T01:12:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_37.export 2024-04-24T01:12:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_37.export 2024-04-24T01:12:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.export 2024-04-24T01:12:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_37.export 2024-04-24T01:12:24.657 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:12:24.653+0000 7efd50ccc640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-24T01:12:24.657 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-04-24T01:12:24.653+0000 7efd52cd0640 -1 bdev(0x559ae0f40e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-24T01:12:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_37.export 2024-04-24T01:12:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_37.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_37.export 2024-04-24T01:12:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:12:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_38 2024-04-24T01:12:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_38 snap 2024-04-24T01:12:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-04-24T01:12:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:12:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:12:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:12:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-04-24T01:12:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:12:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:12:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:12:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:12:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:12:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:12:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:12:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-04-24T01:12:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T01:12:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:12:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-04-24T01:12:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T01:12:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:12:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:12:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:12:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2024-04-24T01:12:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:12:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:12:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:12:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:12:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:12:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:12:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:12:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-04-24T01:12:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_38 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_38 2024-04-24T01:12:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_38 ']' 2024-04-24T01:12:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-04-24T01:12:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-04-24T01:12:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-04-24T01:12:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-04-24T01:12:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T01:12:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-04-24T01:12:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T01:12:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2024-04-24T01:12:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-04-24T01:12:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-04-24T01:12:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-24T01:12:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-24T01:12:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-24T01:12:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814]]], state=connected] 2024-04-24T01:12:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=8816], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814], [object_number=9, tag_tid=1, entry_tid=8813]]], state=connected] 2024-04-24T01:12:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-24T01:12:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-24T01:12:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814]]], state=connected] 2024-04-24T01:12:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=8816], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814], [object_number=9, tag_tid=1, entry_tid=8813]]], state=connected] 2024-04-24T01:12:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=8816' 2024-04-24T01:12:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=8816' 2024-04-24T01:12:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=8816' '!=' '' 2024-04-24T01:12:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:12:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=8816' = '' 2024-04-24T01:12:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=8816' 2024-04-24T01:12:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:12:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=8816' 2024-04-24T01:12:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:12:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:12:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=8816' 2024-04-24T01:12:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=8816 2024-04-24T01:12:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 8816 2024-04-24T01:12:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_38 2024-04-24T01:12:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_38 ']' 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-04-24T01:12:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-04-24T01:12:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T01:12:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-04-24T01:12:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T01:12:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2024-04-24T01:12:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2024-04-24T01:12:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-04-24T01:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-04-24T01:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-24T01:12:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-24T01:12:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-24T01:12:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814]]], state=connected] 2024-04-24T01:12:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=8816], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814], [object_number=9, tag_tid=1, entry_tid=8813]]], state=connected] 2024-04-24T01:12:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-24T01:12:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-24T01:12:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814]]], state=connected] 2024-04-24T01:12:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=8816], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814], [object_number=9, tag_tid=1, entry_tid=8813]]], state=connected] 2024-04-24T01:12:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=8816' 2024-04-24T01:12:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=8816' 2024-04-24T01:12:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=8816' '!=' 'object_number=8, tag_tid=1, entry_tid=8816' 2024-04-24T01:12:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:12:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_38 2024-04-24T01:12:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_38 ']' 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:30.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:30.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-04-24T01:12:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T01:12:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-04-24T01:12:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-04-24T01:12:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T01:12:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_38 2024-04-24T01:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_38 2024-04-24T01:12:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-24T01:12:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-24T01:12:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814]]], state=connected] 2024-04-24T01:12:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814]]], state=connected] 2024-04-24T01:12:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-04-24T01:12:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-04-24T01:12:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:12:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-24T01:12:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-24T01:12:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-24T01:12:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814]]], state=connected] 2024-04-24T01:12:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8815], [object_number=10, tag_tid=1, entry_tid=8814]]], state=connected] 2024-04-24T01:12:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.status 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:12:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-04-24T01:12:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_38': 2024-04-24T01:12:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d8de3119ae 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d8de3119ae 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:03:01 2024 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:03:01 2024 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:04:02 2024 2024-04-24T01:12:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:12:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d8de3119ae 2024-04-24T01:12:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:12:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:12:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 8ba83db3-7e78-4eb4-a232-29e32ee7590c 2024-04-24T01:12:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:12:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_38 snap 2024-04-24T01:12:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:12:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:12:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.export 2024-04-24T01:12:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_38.export 2024-04-24T01:12:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_38.export 2024-04-24T01:12:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.export 2024-04-24T01:12:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_38.export 2024-04-24T01:12:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_38.export 2024-04-24T01:12:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_38.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_38.export 2024-04-24T01:12:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:12:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_39 2024-04-24T01:12:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_39 snap 2024-04-24T01:12:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:12:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:12:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-04-24T01:12:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:12:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:12:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:12:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:12:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-04-24T01:12:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:12:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:12:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:12:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:12:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:12:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:12:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-04-24T01:12:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:12:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:12:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:12:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-04-24T01:12:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-24T01:12:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:12:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:12:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-04-24T01:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-24T01:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-04-24T01:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:12:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-04-24T01:12:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:12:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:12:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:12:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:12:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:12:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:12:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:12:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-04-24T01:12:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_39 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_39 2024-04-24T01:12:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:12:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_39 ']' 2024-04-24T01:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-04-24T01:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-04-24T01:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-04-24T01:12:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-24T01:12:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_39 2024-04-24T01:12:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_39 2024-04-24T01:12:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-24T01:12:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_39 2024-04-24T01:12:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:12:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:12:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:12:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:12:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:12:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:12:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:12:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:12:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:12:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_39 2024-04-24T01:12:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-04-24T01:12:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_39 2024-04-24T01:12:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-04-24T01:12:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_39 2024-04-24T01:12:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-24T01:12:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-24T01:12:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.status 2024-04-24T01:12:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-24T01:12:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.status 2024-04-24T01:12:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9587], [object_number=10, tag_tid=1, entry_tid=9586]]], state=connected] 2024-04-24T01:12:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9587], [object_number=10, tag_tid=1, entry_tid=9586]]], state=connected] 2024-04-24T01:12:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.status 2024-04-24T01:12:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_39 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_39 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-24T01:12:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-24T01:12:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.status 2024-04-24T01:12:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-24T01:12:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.status 2024-04-24T01:12:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9587], [object_number=10, tag_tid=1, entry_tid=9586]]], state=connected] 2024-04-24T01:12:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9587], [object_number=10, tag_tid=1, entry_tid=9586]]], state=connected] 2024-04-24T01:12:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.status 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:12:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-04-24T01:12:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_39': 2024-04-24T01:12:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:12:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:12:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122dad0439f9 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122dad0439f9 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:03:10 2024 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:03:10 2024 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:04:11 2024 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122dad0439f9 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 4aad9ea6-0abe-4fdd-ad48-ed2da2dcf354 2024-04-24T01:12:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:12:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_39 snap 2024-04-24T01:12:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:12:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:12:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.export 2024-04-24T01:12:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_39.export 2024-04-24T01:12:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_39.export 2024-04-24T01:12:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.export 2024-04-24T01:12:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_39.export 2024-04-24T01:12:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_39.export 2024-04-24T01:12:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_39.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_39.export 2024-04-24T01:12:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:12:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_40 2024-04-24T01:12:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_40 snap 2024-04-24T01:12:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:12:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-04-24T01:12:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-04-24T01:12:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:12:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-04-24T01:12:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:12:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:12:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:12:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-04-24T01:12:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:12:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:12:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:12:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:12:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:12:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:12:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:12:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:12:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:12:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-04-24T01:12:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T01:12:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:12:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-04-24T01:12:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:12:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:12:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:12:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:12:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:12:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-04-24T01:12:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:12:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:12:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:12:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:12:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_40 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:12:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:12:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_40 2024-04-24T01:12:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_40 ']' 2024-04-24T01:12:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-24T01:12:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-24T01:12:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-24T01:12:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T01:12:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-24T01:12:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-24T01:12:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-24T01:12:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-24T01:12:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-24T01:12:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=4149' '!=' '' 2024-04-24T01:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=4149' = '' 2024-04-24T01:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:12:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:12:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:12:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=4149 2024-04-24T01:12:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 4149 2024-04-24T01:12:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_40 2024-04-24T01:12:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_40 ']' 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T01:12:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-24T01:12:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-24T01:12:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-24T01:12:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-24T01:12:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=4149' '!=' 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:12:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_40 2024-04-24T01:12:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_40 ']' 2024-04-24T01:12:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-24T01:12:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-24T01:12:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T01:12:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-24T01:12:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-24T01:12:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:43.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=4149' '!=' 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:12:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_40 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_40 ']' 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T01:12:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-24T01:12:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-24T01:12:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-24T01:12:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=4149' '!=' 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:12:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_40 2024-04-24T01:12:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_40 ']' 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T01:12:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-24T01:12:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-24T01:12:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=4149], [object_number=0, tag_tid=1, entry_tid=4148], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=4149' '!=' 'object_number=1, tag_tid=1, entry_tid=4149' 2024-04-24T01:12:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:12:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_40 2024-04-24T01:12:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_40 ']' 2024-04-24T01:12:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-24T01:12:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-24T01:12:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-24T01:12:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T01:12:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:12:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:12:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:12:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:12:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:12:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:12:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:12:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:12:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:12:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-24T01:12:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-24T01:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-24T01:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-24T01:12:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-24T01:12:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:12:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-24T01:12:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-24T01:12:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-24T01:12:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:12:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:12:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4147], [object_number=2, tag_tid=1, entry_tid=4146]]], state=connected] 2024-04-24T01:12:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.status 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=1' -a 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:12:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-04-24T01:12:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_40': 2024-04-24T01:12:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:12:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 123c54ed5251 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.123c54ed5251 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:03:21 2024 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:03:21 2024 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:03:21 2024 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 123c54ed5251 2024-04-24T01:12:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:12:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:12:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 4dc0ce72-662a-423b-806d-f1a81599471e 2024-04-24T01:12:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:12:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_40 snap 2024-04-24T01:12:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:12:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:12:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.export 2024-04-24T01:12:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_40.export 2024-04-24T01:12:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_40.export 2024-04-24T01:12:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.export 2024-04-24T01:12:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_40.export 2024-04-24T01:12:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:12:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_40.export 2024-04-24T01:12:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_40.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_40.export 2024-04-24T01:12:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:12:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_41 2024-04-24T01:12:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_41 snap 2024-04-24T01:12:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:12:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:12:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-04-24T01:12:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:12:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:12:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:12:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-04-24T01:12:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:12:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:12:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:12:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:12:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:12:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:12:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:12:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:12:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:12:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:12:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:12:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:12:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:12:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:12:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:12:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:12:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:12:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-04-24T01:12:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:12:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:12:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:12:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:12:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:12:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:12:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:12:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_41 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:12:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:12:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_41 2024-04-24T01:12:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_41 ']' 2024-04-24T01:12:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-04-24T01:12:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-04-24T01:12:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-04-24T01:12:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:54.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-04-24T01:12:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-04-24T01:12:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10273' '!=' '' 2024-04-24T01:12:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:12:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10273' = '' 2024-04-24T01:12:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:12:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:12:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:12:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:12:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:12:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:12:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10273 2024-04-24T01:12:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 10273 2024-04-24T01:12:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:12:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:12:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_41 2024-04-24T01:12:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_41 ']' 2024-04-24T01:12:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-04-24T01:12:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-04-24T01:12:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-04-24T01:12:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-04-24T01:12:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-04-24T01:12:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-04-24T01:12:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-04-24T01:12:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-04-24T01:12:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10273' '!=' 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_41 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_41 ']' 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-04-24T01:12:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-04-24T01:12:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-04-24T01:12:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10273' '!=' 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:12:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_41 2024-04-24T01:12:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_41 ']' 2024-04-24T01:12:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-04-24T01:12:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-04-24T01:12:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-04-24T01:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-04-24T01:12:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-04-24T01:12:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-04-24T01:12:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10273' '!=' 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_41 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_41 ']' 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:12:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:12:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:12:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:12:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:12:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:12:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:12:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:12:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:12:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:12:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-04-24T01:12:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:12:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-04-24T01:12:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-04-24T01:12:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:12:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:12:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-04-24T01:12:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:12:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:12:58.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:58.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:12:58.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:12:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:12:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:12:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:12:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:12:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10273' '!=' 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:12:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:12:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_41 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_41 ']' 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-04-24T01:13:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-04-24T01:13:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:13:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-04-24T01:13:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:13:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:13:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:13:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:13:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:13:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:13:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-04-24T01:13:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:13:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-04-24T01:13:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:13:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:13:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:13:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:13:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:13:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10273], [object_number=8, tag_tid=1, entry_tid=10272], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:13:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:13:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:13:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10273' '!=' 'object_number=9, tag_tid=1, entry_tid=10273' 2024-04-24T01:13:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:13:02.587 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.32075277266070845) > weight (0.25) 2024-04-24T01:13:02.587 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.46565669906145013) > weight (0.25) 2024-04-24T01:13:02.587 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7202941947271224) > weight (0.25) 2024-04-24T01:13:02.587 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5842936619055712) > weight (0.25) 2024-04-24T01:13:02.587 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-04-24T01:13:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_41 2024-04-24T01:13:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_41 ']' 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-04-24T01:13:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-04-24T01:13:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-04-24T01:13:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:13:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-04-24T01:13:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:13:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:13:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:13:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:13:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:13:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:13:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-04-24T01:13:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:13:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-04-24T01:13:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:13:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-24T01:13:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-24T01:13:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-24T01:13:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:13:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10271], [object_number=10, tag_tid=1, entry_tid=10270]]], state=connected] 2024-04-24T01:13:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.status 2024-04-24T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-04-24T01:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:13:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_41': 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b2863ea576 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b2863ea576 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:13:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:03:28 2024 2024-04-24T01:13:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:03:28 2024 2024-04-24T01:13:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:05:30 2024 2024-04-24T01:13:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:13:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10b2863ea576 2024-04-24T01:13:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:13:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:13:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e10e7910-97e3-4ac8-8c74-526c571d432e 2024-04-24T01:13:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:13:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_41 snap 2024-04-24T01:13:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:13:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:13:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.export 2024-04-24T01:13:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_41.export 2024-04-24T01:13:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_41.export 2024-04-24T01:13:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.export 2024-04-24T01:13:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_41.export 2024-04-24T01:13:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_41.export 2024-04-24T01:13:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_41.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_41.export 2024-04-24T01:13:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:13:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_42 2024-04-24T01:13:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_42 snap 2024-04-24T01:13:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:13:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-04-24T01:13:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-04-24T01:13:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:13:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-04-24T01:13:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:13:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:13:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:13:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-04-24T01:13:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:13:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:13:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:13:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:13:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:13:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:13:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-04-24T01:13:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:13:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:13:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:13:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:13:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-04-24T01:13:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-24T01:13:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:13:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:13:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:13:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-04-24T01:13:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:13:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:13:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:13:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:13:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_42 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:13:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:13:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_42 2024-04-24T01:13:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_42 ']' 2024-04-24T01:13:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-24T01:13:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-24T01:13:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-24T01:13:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-24T01:13:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-24T01:13:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-24T01:13:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-24T01:13:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-24T01:13:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-24T01:13:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-24T01:13:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-24T01:13:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-24T01:13:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7252], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250], [object_number=5, tag_tid=1, entry_tid=7249]]], state=connected] 2024-04-24T01:13:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-24T01:13:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-24T01:13:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-24T01:13:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-24T01:13:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-24T01:13:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7252], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250], [object_number=5, tag_tid=1, entry_tid=7249]]], state=connected] 2024-04-24T01:13:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=7252' '!=' '' 2024-04-24T01:13:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:13:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=7252' = '' 2024-04-24T01:13:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:13:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:13:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:13:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=7252 2024-04-24T01:13:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 7252 2024-04-24T01:13:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_42 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_42 ']' 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-24T01:13:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-24T01:13:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-24T01:13:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-24T01:13:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-24T01:13:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-24T01:13:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-24T01:13:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-24T01:13:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-24T01:13:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7252], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250], [object_number=5, tag_tid=1, entry_tid=7249]]], state=connected] 2024-04-24T01:13:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-24T01:13:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-24T01:13:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-24T01:13:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-24T01:13:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-24T01:13:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:10.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:10.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7252], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250], [object_number=5, tag_tid=1, entry_tid=7249]]], state=connected] 2024-04-24T01:13:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=7252' '!=' 'object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:13:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_42 2024-04-24T01:13:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_42 ']' 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-24T01:13:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-24T01:13:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-24T01:13:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-24T01:13:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-24T01:13:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-24T01:13:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-24T01:13:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-24T01:13:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-24T01:13:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-24T01:13:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7252], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250], [object_number=5, tag_tid=1, entry_tid=7249]]], state=connected] 2024-04-24T01:13:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-24T01:13:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-24T01:13:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-24T01:13:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-24T01:13:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-24T01:13:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7252], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250], [object_number=5, tag_tid=1, entry_tid=7249]]], state=connected] 2024-04-24T01:13:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=7252' '!=' 'object_number=4, tag_tid=1, entry_tid=7252' 2024-04-24T01:13:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_42 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_42 ']' 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-24T01:13:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-24T01:13:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-24T01:13:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-24T01:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-24T01:13:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-24T01:13:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-24T01:13:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-24T01:13:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-24T01:13:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-24T01:13:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:13:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-24T01:13:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-24T01:13:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-24T01:13:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7251], [object_number=6, tag_tid=1, entry_tid=7250]]], state=connected] 2024-04-24T01:13:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.status 2024-04-24T01:13:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:13:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-04-24T01:13:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_42': 2024-04-24T01:13:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:13:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:13:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:13:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d8495dab35 2024-04-24T01:13:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d8495dab35 2024-04-24T01:13:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:13:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:13:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:13:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:13:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:03:41 2024 2024-04-24T01:13:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:03:41 2024 2024-04-24T01:13:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:03:41 2024 2024-04-24T01:13:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:13:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d8495dab35 2024-04-24T01:13:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:13:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:13:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 07c2a847-221b-4f0e-998a-6387fca6159a 2024-04-24T01:13:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:13:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_42 snap 2024-04-24T01:13:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:13:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:13:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.export 2024-04-24T01:13:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_42.export 2024-04-24T01:13:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_42.export 2024-04-24T01:13:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.export 2024-04-24T01:13:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_42.export 2024-04-24T01:13:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_42.export 2024-04-24T01:13:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_42.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_42.export 2024-04-24T01:13:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:13:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_43 2024-04-24T01:13:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_43 snap 2024-04-24T01:13:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:13:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-04-24T01:13:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-04-24T01:13:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:13:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-04-24T01:13:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:13:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:13:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:13:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-04-24T01:13:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:13:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:13:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:13:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:13:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:13:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:13:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-04-24T01:13:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:13:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:13:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:13:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:13:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:13:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-04-24T01:13:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:13:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-04-24T01:13:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:13:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:13:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-04-24T01:13:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:13:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-04-24T01:13:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:13:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:13:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:13:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:13:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:13:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:13:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:13:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-04-24T01:13:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_43 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_43 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_43 ']' 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-04-24T01:13:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2024-04-24T01:13:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-04-24T01:13:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-04-24T01:13:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-04-24T01:13:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-04-24T01:13:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-24T01:13:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:13:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-24T01:13:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:13:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T01:13:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:13:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=1 2024-04-24T01:13:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-24T01:13:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_43 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_43 ']' 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2024-04-24T01:13:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2024-04-24T01:13:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-04-24T01:13:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-04-24T01:13:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-04-24T01:13:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-04-24T01:13:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:13:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_43 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_43 ']' 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2024-04-24T01:13:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-04-24T01:13:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-04-24T01:13:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-04-24T01:13:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-04-24T01:13:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:13:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_43 2024-04-24T01:13:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_43 ']' 2024-04-24T01:13:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-04-24T01:13:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-04-24T01:13:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-04-24T01:13:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-04-24T01:13:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_43 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_43 ']' 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2024-04-24T01:13:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T01:13:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_43 2024-04-24T01:13:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_43 2024-04-24T01:13:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-04-24T01:13:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-04-24T01:13:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-04-24T01:13:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-04-24T01:13:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:13:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-24T01:13:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-24T01:13:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-24T01:13:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5471], [object_number=6, tag_tid=1, entry_tid=5470]]], state=connected] 2024-04-24T01:13:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.status 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:13:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-04-24T01:13:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_43': 2024-04-24T01:13:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:13:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:13:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:13:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122d61e72df1 2024-04-24T01:13:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122d61e72df1 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:03:53 2024 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:03:53 2024 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:04:54 2024 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122d61e72df1 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: bf6e00e6-fcb3-4606-98d9-9bba25bbed25 2024-04-24T01:13:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:13:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_43 snap 2024-04-24T01:13:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:13:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:13:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.export 2024-04-24T01:13:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_43.export 2024-04-24T01:13:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_43.export 2024-04-24T01:13:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.export 2024-04-24T01:13:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_43.export 2024-04-24T01:13:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_43.export 2024-04-24T01:13:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_43.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_43.export 2024-04-24T01:13:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:13:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_44 2024-04-24T01:13:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_44 snap 2024-04-24T01:13:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:13:24.353 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:13:24.351+0000 7f6307f66640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:13:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-04-24T01:13:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:13:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:13:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:13:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-04-24T01:13:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:13:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:13:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:13:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:13:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:13:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:13:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:13:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:13:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-04-24T01:13:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:13:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:13:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:13:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:13:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_44 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:13:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:13:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_44 2024-04-24T01:13:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_44 ']' 2024-04-24T01:13:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-24T01:13:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-24T01:13:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-24T01:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-24T01:13:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-24T01:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-24T01:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=3036], [object_number=3, tag_tid=1, entry_tid=3035], [object_number=2, tag_tid=1, entry_tid=3034], [object_number=1, tag_tid=1, entry_tid=3033]]], state=connected] 2024-04-24T01:13:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=3036], [object_number=3, tag_tid=1, entry_tid=3035], [object_number=2, tag_tid=1, entry_tid=3034], [object_number=1, tag_tid=1, entry_tid=3033]]], state=connected] 2024-04-24T01:13:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=0, tag_tid=1, entry_tid=3036' '!=' '' 2024-04-24T01:13:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:13:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=0, tag_tid=1, entry_tid=3036' = '' 2024-04-24T01:13:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:13:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:13:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:13:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3036 2024-04-24T01:13:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 1 -a 1 -le 3036 2024-04-24T01:13:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_44 2024-04-24T01:13:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_44 ']' 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-24T01:13:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-24T01:13:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:26.982 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.803407078562144) > weight (0.25) 2024-04-24T01:13:26.982 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8628552831364484) > weight (0.25) 2024-04-24T01:13:26.982 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-24T01:13:26.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-24T01:13:26.983 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T01:13:26.983 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-24T01:13:26.983 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:13:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-24T01:13:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-24T01:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:26.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:13:26.983+0000 7f842e791640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 102276) UID: 1000 2024-04-24T01:13:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=3036], [object_number=3, tag_tid=1, entry_tid=3035], [object_number=2, tag_tid=1, entry_tid=3034], [object_number=1, tag_tid=1, entry_tid=3033]]], state=connected] 2024-04-24T01:13:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-24T01:13:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-24T01:13:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=3036], [object_number=3, tag_tid=1, entry_tid=3035], [object_number=2, tag_tid=1, entry_tid=3034], [object_number=1, tag_tid=1, entry_tid=3033]]], state=connected] 2024-04-24T01:13:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=0, tag_tid=1, entry_tid=3036' '!=' 'object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:13:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_44 2024-04-24T01:13:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_44 ']' 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-24T01:13:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:13:27.647+0000 7f727705c640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x557b30b10700 handle_flush_local_replay: error flushing local replay: (108) Cannot send after transport endpoint shutdown 2024-04-24T01:13:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-24T01:13:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-24T01:13:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=3036], [object_number=3, tag_tid=1, entry_tid=3035], [object_number=2, tag_tid=1, entry_tid=3034], [object_number=1, tag_tid=1, entry_tid=3033]]], state=connected] 2024-04-24T01:13:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-24T01:13:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-24T01:13:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=3036], [object_number=3, tag_tid=1, entry_tid=3035], [object_number=2, tag_tid=1, entry_tid=3034], [object_number=1, tag_tid=1, entry_tid=3033]]], state=connected] 2024-04-24T01:13:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=0, tag_tid=1, entry_tid=3036' '!=' 'object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:13:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_44 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_44 ']' 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-24T01:13:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-24T01:13:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-24T01:13:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3249], [object_number=0, tag_tid=1, entry_tid=3248], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-24T01:13:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-24T01:13:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3249], [object_number=0, tag_tid=1, entry_tid=3248], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=3249' 2024-04-24T01:13:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=3249' 2024-04-24T01:13:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3249' '!=' 'object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:13:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3249' = 'object_number=0, tag_tid=1, entry_tid=3036' 2024-04-24T01:13:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=3249' 2024-04-24T01:13:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:13:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=3249' 2024-04-24T01:13:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:13:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:13:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=3249' 2024-04-24T01:13:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=3249 2024-04-24T01:13:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 1 -a 1 -le 3249 2024-04-24T01:13:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_44 2024-04-24T01:13:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_44 ']' 2024-04-24T01:13:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-24T01:13:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-24T01:13:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-24T01:13:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-24T01:13:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-24T01:13:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-24T01:13:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-24T01:13:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-24T01:13:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:13:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-24T01:13:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-24T01:13:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-24T01:13:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 0 2024-04-24T01:13:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 0 2024-04-24T01:13:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3247], [object_number=2, tag_tid=1, entry_tid=3246]]], state=connected] 2024-04-24T01:13:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.status 2024-04-24T01:13:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-24T01:13:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-04-24T01:13:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:13:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-04-24T01:13:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_44': 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1256c6e063d1 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1256c6e063d1 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:03:59 2024 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:03:59 2024 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:10:15 2024 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:13:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 1256c6e063d1 2024-04-24T01:13:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:13:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:13:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 6e237009-33ce-4729-b6bc-1e0212c1762c 2024-04-24T01:13:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:13:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_44 snap 2024-04-24T01:13:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:13:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:13:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.export 2024-04-24T01:13:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_44.export 2024-04-24T01:13:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_44.export 2024-04-24T01:13:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.export 2024-04-24T01:13:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_44.export 2024-04-24T01:13:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_44.export 2024-04-24T01:13:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_44.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_44.export 2024-04-24T01:13:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:13:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_45 2024-04-24T01:13:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_45 snap 2024-04-24T01:13:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:13:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-04-24T01:13:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-04-24T01:13:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:13:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-04-24T01:13:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:13:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:13:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:13:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-04-24T01:13:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:13:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:13:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:13:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:13:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:13:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:13:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:13:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:13:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:13:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-04-24T01:13:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:13:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-04-24T01:13:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:13:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:13:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:33.085 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-24T01:13:33.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-24T01:13:33.085 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-24T01:13:33.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-24T01:13:33.087 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-04-24T01:13:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-04-24T01:13:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:13:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:13:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:13:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-04-24T01:13:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:13:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2024-04-24T01:13:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:13:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:13:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2024-04-24T01:13:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:13:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:13:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:13:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:13:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_45 2024-04-24T01:13:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:13:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:13:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_45 2024-04-24T01:13:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_45 ']' 2024-04-24T01:13:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-24T01:13:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-24T01:13:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-24T01:13:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_45 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:34.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:34.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:34.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_45 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-24T01:13:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-24T01:13:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-24T01:13:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-24T01:13:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-24T01:13:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:13:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-24T01:13:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=3 2024-04-24T01:13:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=2 2024-04-24T01:13:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:13:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=1 2024-04-24T01:13:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-24T01:13:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_45 2024-04-24T01:13:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_45 ']' 2024-04-24T01:13:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-24T01:13:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-24T01:13:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-24T01:13:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-24T01:13:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_45 2024-04-24T01:13:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_45 2024-04-24T01:13:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-24T01:13:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-24T01:13:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:13:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_45 2024-04-24T01:13:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_45 ']' 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_45 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_45 2024-04-24T01:13:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-24T01:13:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-24T01:13:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:13:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_45 2024-04-24T01:13:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_45 ']' 2024-04-24T01:13:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-24T01:13:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-24T01:13:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-24T01:13:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_45 2024-04-24T01:13:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_45 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-24T01:13:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-24T01:13:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-24T01:13:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-24T01:13:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_45 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_45 ']' 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-24T01:13:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-24T01:13:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_45 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_45 2024-04-24T01:13:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-24T01:13:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-24T01:13:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-24T01:13:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-24T01:13:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:13:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:13:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_45 2024-04-24T01:13:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_45 ']' 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-24T01:13:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_45 2024-04-24T01:13:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T01:13:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_45 2024-04-24T01:13:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_45 2024-04-24T01:13:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-24T01:13:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-24T01:13:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-24T01:13:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-24T01:13:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:13:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:13:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7183], [object_number=6, tag_tid=1, entry_tid=7182]]], state=connected] 2024-04-24T01:13:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.status 2024-04-24T01:13:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-24T01:13:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-04-24T01:13:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:13:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-04-24T01:13:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_45': 2024-04-24T01:13:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:13:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:13:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:13:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1256713bbe0f 2024-04-24T01:13:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1256713bbe0f 2024-04-24T01:13:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:04:07 2024 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:04:07 2024 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:05:26 2024 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 1256713bbe0f 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 2328fc4a-63e7-42d6-9e4c-862ea4377e4f 2024-04-24T01:13:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:13:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_45 snap 2024-04-24T01:13:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:13:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:13:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.export 2024-04-24T01:13:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_45.export 2024-04-24T01:13:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_45.export 2024-04-24T01:13:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.export 2024-04-24T01:13:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_45.export 2024-04-24T01:13:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_45.export 2024-04-24T01:13:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_45.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_45.export 2024-04-24T01:13:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:13:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_46 2024-04-24T01:13:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_46 snap 2024-04-24T01:13:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:13:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-04-24T01:13:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-04-24T01:13:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:13:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:13:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:13:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-04-24T01:13:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:13:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:13:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:13:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:13:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:13:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:13:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:13:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-04-24T01:13:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:13:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-04-24T01:13:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:13:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:13:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-04-24T01:13:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:13:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:13:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:13:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-04-24T01:13:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:13:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2024-04-24T01:13:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:13:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:13:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:13:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:13:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:13:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:13:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:13:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:13:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2024-04-24T01:13:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:13:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:13:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:13:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:13:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:13:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:13:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:13:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_46 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:13:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:13:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_46 2024-04-24T01:13:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_46 ']' 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-24T01:13:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-24T01:13:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2024-04-24T01:13:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_46 2024-04-24T01:13:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-24T01:13:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-24T01:13:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-24T01:13:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-24T01:13:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10425' '!=' '' 2024-04-24T01:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10425' = '' 2024-04-24T01:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:13:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:13:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:13:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:13:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:13:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:13:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=10425 2024-04-24T01:13:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 10425 2024-04-24T01:13:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:13:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_46 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_46 ']' 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-24T01:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2024-04-24T01:13:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_46 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-24T01:13:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-24T01:13:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-24T01:13:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-24T01:13:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10425' '!=' 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:13:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_46 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_46 ']' 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2024-04-24T01:13:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_46 2024-04-24T01:13:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-24T01:13:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-24T01:13:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-24T01:13:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-24T01:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10425' '!=' 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:13:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_46 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_46 ']' 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-24T01:13:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-24T01:13:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2024-04-24T01:13:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_46 2024-04-24T01:13:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-24T01:13:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-24T01:13:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-24T01:13:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-24T01:13:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10425' '!=' 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_46 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_46 ']' 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_46 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-24T01:13:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-24T01:13:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-24T01:13:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-24T01:13:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10425], [object_number=8, tag_tid=1, entry_tid=10424], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10425' '!=' 'object_number=9, tag_tid=1, entry_tid=10425' 2024-04-24T01:13:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:13:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:13:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_46 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_46 ']' 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:13:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:13:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:13:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:13:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:13:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-24T01:13:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:13:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:13:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:13:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:13:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:13:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:13:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:13:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:13:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-24T01:13:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:13:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:13:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:13:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:13:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:13:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:13:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:13:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:13:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:13:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:13:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:13:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:13:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:13:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:13:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:13:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_46 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:13:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:13:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:13:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:13:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:13:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:13:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:13:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T01:13:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:13:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_46 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:13:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:13:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:13:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:13:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:13:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:13:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:13:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:13:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:13:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_46 2024-04-24T01:13:58.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-24T01:13:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-24T01:13:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:13:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:58.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:58.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-24T01:13:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-24T01:13:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:13:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:13:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:13:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:13:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-24T01:13:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-24T01:13:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:13:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-24T01:13:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:13:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:13:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:13:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10423], [object_number=10, tag_tid=1, entry_tid=10422]]], state=connected] 2024-04-24T01:13:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.status 2024-04-24T01:13:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:13:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-04-24T01:13:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:13:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:13:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:13:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:13:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-04-24T01:13:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_46': 2024-04-24T01:13:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:13:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:13:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:13:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d8185b5e69 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d8185b5e69 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:04:16 2024 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:04:16 2024 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:05:16 2024 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d8185b5e69 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 453af0ae-40f6-40cb-8e54-50012e44d7b1 2024-04-24T01:13:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:13:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_46 snap 2024-04-24T01:13:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:13:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:13:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.export 2024-04-24T01:13:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_46.export 2024-04-24T01:13:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_46.export 2024-04-24T01:13:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.export 2024-04-24T01:13:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_46.export 2024-04-24T01:13:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:13:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_46.export 2024-04-24T01:13:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_46.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_46.export 2024-04-24T01:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_47 2024-04-24T01:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_47 snap 2024-04-24T01:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:14:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-04-24T01:14:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-04-24T01:14:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:14:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:14:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:14:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-04-24T01:14:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:14:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:14:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:14:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-04-24T01:14:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:14:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:14:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:14:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:14:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:14:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:14:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-04-24T01:14:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:14:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:14:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:14:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:14:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:14:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-04-24T01:14:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-04-24T01:14:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:14:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:14:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:14:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-04-24T01:14:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:14:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:14:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:14:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:14:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-04-24T01:14:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-04-24T01:14:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:14:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:14:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:14:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:14:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:14:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-04-24T01:14:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:14:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:14:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:14:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:14:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:14:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:14:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:14:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-04-24T01:14:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:14:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_47 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:14:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_47 2024-04-24T01:14:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_47 ']' 2024-04-24T01:14:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-04-24T01:14:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-04-24T01:14:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2024-04-24T01:14:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2024-04-24T01:14:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_47 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-04-24T01:14:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-04-24T01:14:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9713], [object_number=8, tag_tid=1, entry_tid=9712], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9713], [object_number=8, tag_tid=1, entry_tid=9712], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9713' '!=' '' 2024-04-24T01:14:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:14:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9713' = '' 2024-04-24T01:14:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:14:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:14:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:14:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:14:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:14:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:14:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:14:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:14:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:14:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=9713 2024-04-24T01:14:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 9713 2024-04-24T01:14:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:14:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:14:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_47 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_47 ']' 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-04-24T01:14:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2024-04-24T01:14:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_47 2024-04-24T01:14:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:14:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:14:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-04-24T01:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-04-24T01:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9713], [object_number=8, tag_tid=1, entry_tid=9712], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:14:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-04-24T01:14:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-04-24T01:14:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9713], [object_number=8, tag_tid=1, entry_tid=9712], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9713' '!=' 'object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_47 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_47 ']' 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2024-04-24T01:14:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2024-04-24T01:14:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_47 2024-04-24T01:14:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:14:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:14:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-04-24T01:14:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-04-24T01:14:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9713], [object_number=8, tag_tid=1, entry_tid=9712], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9713], [object_number=8, tag_tid=1, entry_tid=9712], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9713' '!=' 'object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:14:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_47 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_47 ']' 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-04-24T01:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2024-04-24T01:14:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2024-04-24T01:14:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_47 2024-04-24T01:14:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:14:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:14:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:14:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:14:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-04-24T01:14:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-04-24T01:14:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9713], [object_number=8, tag_tid=1, entry_tid=9712], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9713], [object_number=8, tag_tid=1, entry_tid=9712], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9713' '!=' 'object_number=9, tag_tid=1, entry_tid=9713' 2024-04-24T01:14:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:14:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_47 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_47 ']' 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2024-04-24T01:14:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_47 2024-04-24T01:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T01:14:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/image_47 2024-04-24T01:14:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:14:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T01:14:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T01:14:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T01:14:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:3 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/image_47 2024-04-24T01:14:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-04-24T01:14:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-04-24T01:14:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-24T01:14:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-24T01:14:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-24T01:14:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 2 2024-04-24T01:14:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 2 2024-04-24T01:14:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9711], [object_number=10, tag_tid=1, entry_tid=9710]]], state=connected] 2024-04-24T01:14:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.status 2024-04-24T01:14:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:14:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-24T01:14:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-04-24T01:14:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:14:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:14:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:14:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:14:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-04-24T01:14:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_47': 2024-04-24T01:14:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:14:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:14:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:14:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1256ae0ea7e1 2024-04-24T01:14:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1256ae0ea7e1 2024-04-24T01:14:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:14:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:04:26 2024 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:04:26 2024 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:05:27 2024 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 1256ae0ea7e1 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 0d631170-3b30-4bb6-802c-287de471576a 2024-04-24T01:14:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:14:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_47 snap 2024-04-24T01:14:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:14:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:14:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:14:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.export 2024-04-24T01:14:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_47.export 2024-04-24T01:14:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_47.export 2024-04-24T01:14:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.export 2024-04-24T01:14:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:14:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_47.export 2024-04-24T01:14:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:14:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_47.export 2024-04-24T01:14:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_47.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_47.export 2024-04-24T01:14:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:14:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_48 2024-04-24T01:14:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_48 snap 2024-04-24T01:14:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:14:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:14:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-04-24T01:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:14:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:14:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-04-24T01:14:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:14:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:14:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:14:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:14:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:14:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:14:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:14:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-04-24T01:14:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:14:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:14:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-04-24T01:14:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:14:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:14:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-04-24T01:14:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:14:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:14:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:14:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:14:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_48 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:14:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:14:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:14:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:14:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_48 2024-04-24T01:14:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_48 ']' 2024-04-24T01:14:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-24T01:14:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-24T01:14:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-24T01:14:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2024-04-24T01:14:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-24T01:14:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-24T01:14:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=12498], [object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:14:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-24T01:14:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=12498], [object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=12498' '!=' '' 2024-04-24T01:14:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:14:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=12498' = '' 2024-04-24T01:14:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:14:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:14:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:14:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:14:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:14:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:14:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:14:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:14:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:14:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=12498 2024-04-24T01:14:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 12498 2024-04-24T01:14:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:14:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:14:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_48 2024-04-24T01:14:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_48 ']' 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2024-04-24T01:14:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-24T01:14:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-24T01:14:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=12498], [object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=12498], [object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=12498' '!=' 'object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:14:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_48 2024-04-24T01:14:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_48 ']' 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:19.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:19.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-24T01:14:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-24T01:14:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-24T01:14:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-24T01:14:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=12498], [object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:14:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-24T01:14:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-24T01:14:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=12498], [object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=12498' '!=' 'object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:14:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_48 2024-04-24T01:14:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_48 ']' 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-24T01:14:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-24T01:14:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=12498], [object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:14:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-24T01:14:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=12498], [object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=12498' '!=' 'object_number=14, tag_tid=1, entry_tid=12498' 2024-04-24T01:14:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_48 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_48 ']' 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T01:14:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_48 2024-04-24T01:14:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_48 2024-04-24T01:14:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-24T01:14:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-24T01:14:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:14:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-24T01:14:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-24T01:14:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:14:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-24T01:14:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-24T01:14:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-24T01:14:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 3 2024-04-24T01:14:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 3 2024-04-24T01:14:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12498], [object_number=15, tag_tid=1, entry_tid=12495]]], state=connected] 2024-04-24T01:14:23.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.status 2024-04-24T01:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-24T01:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-04-24T01:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:14:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:14:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:14:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-04-24T01:14:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_48': 2024-04-24T01:14:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:14:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:14:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:14:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12568de0e421 2024-04-24T01:14:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12568de0e421 2024-04-24T01:14:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:14:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:04:39 2024 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:04:39 2024 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:05:40 2024 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 12568de0e421 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 50ca65c5-e244-4f9c-955d-0d365f8f0718 2024-04-24T01:14:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:14:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_48 snap 2024-04-24T01:14:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:14:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:14:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:14:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.export 2024-04-24T01:14:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_48.export 2024-04-24T01:14:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_48.export 2024-04-24T01:14:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.export 2024-04-24T01:14:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:14:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_48.export 2024-04-24T01:14:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:14:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_48.export 2024-04-24T01:14:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_48.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_48.export 2024-04-24T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_49 2024-04-24T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_49 snap 2024-04-24T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:14:25.588 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.32016764002624687) > weight (0.25) 2024-04-24T01:14:25.589 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T01:14:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T01:14:25.589 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T01:14:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:14:25.589+0000 7f56e8df3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 116439) UID: 1000 2024-04-24T01:14:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-04-24T01:14:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:14:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-04-24T01:14:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:14:29.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:14:29.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:14:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-04-24T01:14:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:14:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:14:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:14:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:14:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:14:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:14:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-04-24T01:14:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:14:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:14:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-04-24T01:14:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-24T01:14:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-04-24T01:14:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:14:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:14:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:14:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-04-24T01:14:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T01:14:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:14:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:14:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:14:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-04-24T01:14:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:14:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:14:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:14:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:14:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:14:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_49 2024-04-24T01:14:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:14:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:14:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:14:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:14:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_49 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_49 ']' 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-24T01:14:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-24T01:14:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T01:14:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_49 2024-04-24T01:14:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:30.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6853], [object_number=4, tag_tid=1, entry_tid=6852], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6853], [object_number=4, tag_tid=1, entry_tid=6852], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6853' '!=' '' 2024-04-24T01:14:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:14:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6853' = '' 2024-04-24T01:14:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:14:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:14:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:14:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:14:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:14:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:14:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:14:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6853 2024-04-24T01:14:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 6853 2024-04-24T01:14:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:14:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:14:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_49 2024-04-24T01:14:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_49 ']' 2024-04-24T01:14:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-24T01:14:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-24T01:14:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-24T01:14:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T01:14:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_49 2024-04-24T01:14:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-24T01:14:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-24T01:14:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6853], [object_number=4, tag_tid=1, entry_tid=6852], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6853], [object_number=4, tag_tid=1, entry_tid=6852], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6853' '!=' 'object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:14:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_49 2024-04-24T01:14:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_49 ']' 2024-04-24T01:14:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-24T01:14:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-24T01:14:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-24T01:14:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-24T01:14:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T01:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_49 2024-04-24T01:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-24T01:14:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-24T01:14:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6853], [object_number=4, tag_tid=1, entry_tid=6852], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-24T01:14:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-24T01:14:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6853], [object_number=4, tag_tid=1, entry_tid=6852], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6853' '!=' 'object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_49 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_49 ']' 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-24T01:14:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-24T01:14:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:32.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T01:14:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_49 2024-04-24T01:14:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-24T01:14:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-24T01:14:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6853], [object_number=4, tag_tid=1, entry_tid=6852], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-24T01:14:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-24T01:14:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6853], [object_number=4, tag_tid=1, entry_tid=6852], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6853' '!=' 'object_number=5, tag_tid=1, entry_tid=6853' 2024-04-24T01:14:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:14:33.590 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T01:14:33.590 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:14:33.590 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T01:14:33.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T01:14:33.590 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T01:14:33.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T01:14:33.593 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-04-24T01:14:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_49 2024-04-24T01:14:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_49 ']' 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-24T01:14:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-24T01:14:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-24T01:14:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/image_49 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T01:14:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T01:14:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:2 2024-04-24T01:14:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/image_49 2024-04-24T01:14:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-24T01:14:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-24T01:14:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:35.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-24T01:14:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-24T01:14:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-24T01:14:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6851], [object_number=6, tag_tid=1, entry_tid=6850]]], state=connected] 2024-04-24T01:14:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.status 2024-04-24T01:14:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-04-24T01:14:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:14:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:14:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:14:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:14:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-04-24T01:14:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_49': 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10d84a3bbd06 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10d84a3bbd06 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:04:50 2024 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:04:50 2024 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:04:50 2024 2024-04-24T01:14:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:14:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 10d84a3bbd06 2024-04-24T01:14:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:14:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:14:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: c9d20bde-7a50-4f8b-9c54-900085034486 2024-04-24T01:14:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:14:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_49 snap 2024-04-24T01:14:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:14:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:14:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:14:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.export 2024-04-24T01:14:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_49.export 2024-04-24T01:14:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_49.export 2024-04-24T01:14:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.export 2024-04-24T01:14:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:14:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_49.export 2024-04-24T01:14:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:14:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_49.export 2024-04-24T01:14:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_49.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_49.export 2024-04-24T01:14:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:14:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_50 2024-04-24T01:14:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_50 snap 2024-04-24T01:14:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:14:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:14:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T01:14:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-04-24T01:14:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-04-24T01:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T01:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T01:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T01:14:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-04-24T01:14:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T01:14:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T01:14:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T01:14:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T01:14:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T01:14:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:14:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:14:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-04-24T01:14:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-04-24T01:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:14:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:14:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:40.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:40.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-04-24T01:14:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-04-24T01:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-04-24T01:14:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:14:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:14:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2024-04-24T01:14:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 1 2024-04-24T01:14:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:14:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-04-24T01:14:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:14:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:14:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:14:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:14:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-04-24T01:14:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-04-24T01:14:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:14:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:14:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:14:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-04-24T01:14:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:14:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:14:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-04-24T01:14:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2024-04-24T01:14:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:14:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:14:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:14:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2024-04-24T01:14:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 2 2024-04-24T01:14:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:14:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-04-24T01:14:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:14:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:14:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-04-24T01:14:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:14:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:14:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-04-24T01:14:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-04-24T01:14:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:14:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:14:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:14:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-04-24T01:14:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:14:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:14:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:14:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:14:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2024-04-24T01:14:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 4 2024-04-24T01:14:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:14:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-04-24T01:14:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:14:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:14:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:14:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-04-24T01:14:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:14:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:14:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-04-24T01:14:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-04-24T01:14:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T01:14:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T01:14:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T01:14:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T01:14:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T01:14:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T01:14:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T01:14:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T01:14:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T01:14:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T01:14:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-04-24T01:14:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T01:14:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T01:14:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T01:14:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T01:14:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2024-04-24T01:14:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 8 2024-04-24T01:14:56.012 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:14:56.004+0000 7fbdbe182640 -1 reset not still connected to 0x5622d0dbf1e0 2024-04-24T01:14:56.012 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:14:56.004+0000 7fbdbe182640 -1 reset not still connected to 0x5622d2201930 2024-04-24T01:14:56.012 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:14:56.004+0000 7fbdbe182640 -1 reset not still connected to 0x5622e890a750 2024-04-24T01:14:56.016 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:14:56.016+0000 7fbdbe182640 -1 reset not still connected to 0x5622ec6b3e10 2024-04-24T01:14:56.016 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:14:56.016+0000 7fbdbe182640 -1 reset not still connected to 0x5622f1422c30 2024-04-24T01:14:56.017 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:14:56.016+0000 7fbdbe182640 -1 reset not still connected to 0x5622feea49c0 2024-04-24T01:14:56.017 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:14:56.016+0000 7fbdbe182640 -1 reset not still connected to 0x56230d133d40 2024-04-24T01:14:57.240 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:14:57.240+0000 7f6301759640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 18 2024-04-24T01:14:57.243 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:14:57.240+0000 7f6307765640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 29 2024-04-24T01:14:57.245 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:14:57.244+0000 7f6307f66640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 24 2024-04-24T01:14:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T01:14:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-04-24T01:14:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T01:14:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T01:14:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-04-24T01:14:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T01:14:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-04-24T01:14:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T01:14:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T01:14:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T01:14:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T01:14:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T01:14:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T01:14:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T01:14:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T01:14:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T01:14:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T01:14:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T01:14:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T01:14:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T01:14:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-04-24T01:14:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T01:14:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T01:14:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T01:14:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T01:14:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T01:14:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T01:14:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T01:14:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' journal = journal ']' 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_50 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-24T01:14:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:14:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:14:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_50 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_50 ']' 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-04-24T01:14:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-04-24T01:14:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-04-24T01:14:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:14:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:14:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:14:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:14:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:14:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=6738' '!=' '' 2024-04-24T01:14:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ break 2024-04-24T01:14:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=6738' = '' 2024-04-24T01:14:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_mirror_pos='object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:14:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_tag=2 2024-04-24T01:14:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-24T01:14:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_tag=1 2024-04-24T01:14:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:14:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_entry=1 2024-04-24T01:14:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-24T01:14:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut -d= -f 2 2024-04-24T01:14:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_entry=6738 2024-04-24T01:14:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 = 1 -a 1 -le 6738 2024-04-24T01:14:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:14:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_50 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_50 ']' 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-04-24T01:14:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-04-24T01:14:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-04-24T01:14:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-04-24T01:14:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-04-24T01:14:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:14:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:14:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:14:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-04-24T01:14:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-04-24T01:14:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:14:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:14:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:14:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=6738' '!=' 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T01:14:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_50 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_50 ']' 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:14:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:14:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:14:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:14:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:14:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:14:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:14:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:14:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:14:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:14:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:14:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-04-24T01:14:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:14:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:14:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:14:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:14:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-04-24T01:14:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:14:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:14:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:14:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:14:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:14:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:14:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-04-24T01:14:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:14:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:14:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:14:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:14:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:14:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:14:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:14:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:14:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:14:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:14:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=6738' '!=' 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:14:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:14:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T01:15:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_50 2024-04-24T01:15:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:15:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:15:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:15:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_50 ']' 2024-04-24T01:15:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-04-24T01:15:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-04-24T01:15:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:15:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:15:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:15:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:15:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:15:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:15:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:15:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:15:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-04-24T01:15:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:15:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:15:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:15:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:15:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:15:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:15:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:15:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:15:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:15:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:15:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-04-24T01:15:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:15:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:15:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:15:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:15:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-04-24T01:15:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:15:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:15:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:15:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:15:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:15:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:15:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:15:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-04-24T01:15:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-04-24T01:15:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-04-24T01:15:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:15:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:15:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:15:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:15:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:15:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:15:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:15:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:15:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:15:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-04-24T01:15:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-04-24T01:15:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:15:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:15:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:15:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:15:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:15:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:15:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:15:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:15:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:15:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:15:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=6738' '!=' 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:15:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:15:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T01:15:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_50 2024-04-24T01:15:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:15:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_50 ']' 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:15:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:15:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:15:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:15:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-04-24T01:15:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:15:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:15:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:15:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:15:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:15:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:15:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:15:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-04-24T01:15:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:15:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:15:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:15:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:15:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-04-24T01:15:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:15:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:15:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:15:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:15:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:15:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:15:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:15:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-04-24T01:15:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:15:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:15:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:15:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:15:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:15:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:15:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:15:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:15:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:15:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:15:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:15:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:15:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:15:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:15:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:15:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:15:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:15:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=6738' '!=' 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:15:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:15:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_50 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_50 ']' 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:15:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:15:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:15:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:15:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:15:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:15:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:15:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-04-24T01:15:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:15:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:15:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:15:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:15:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:15:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:15:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:15:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:15:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:15:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:15:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:15:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:15:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:15:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:15:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:15:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:15:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:15:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:15:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-04-24T01:15:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:15:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:15:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:15:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:15:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-04-24T01:15:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:15:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:15:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:15:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:15:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:15:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:15:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:15:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:15:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:15:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:15:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:15:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:15:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:15:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:15:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:15:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:15:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:15:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:15:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-04-24T01:15:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-04-24T01:15:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-04-24T01:15:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:15:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:15:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:15:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:15:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:15:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:15:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:15:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:15:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-04-24T01:15:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-04-24T01:15:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:15:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:15:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:15:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:15:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:15:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:15:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:15:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:15:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6738], [object_number=5, tag_tid=1, entry_tid=6737], [object_number=4, tag_tid=1, entry_tid=6736], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:15:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:15:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=6738' '!=' 'object_number=6, tag_tid=1, entry_tid=6738' 2024-04-24T01:15:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T01:15:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T01:15:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ flush cluster1 mirror image_50 2024-04-24T01:15:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:15:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:15:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'cmd=rbd mirror flush' 2024-04-24T01:15:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n image_50 ']' 2024-04-24T01:15:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-04-24T01:15:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-04-24T01:15:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_instance=cluster1 2024-04-24T01:15:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:15:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance=cluster1 2024-04-24T01:15:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loop_instance 2024-04-24T01:15:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T01:15:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-24T01:15:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:15:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:15:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-04-24T01:15:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:15:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:15:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:0 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:15:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T01:15:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:15:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:15:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T01:15:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T01:15:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T01:15:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T01:15:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-04-24T01:15:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T01:15:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror flush : flush rbd mirror 2024-04-24T01:15:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T01:15:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:15:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-04-24T01:15:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:15:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:15:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:15:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T01:15:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T01:15:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:15:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:15:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:15:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T01:15:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T01:15:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:15:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T01:15:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local local_cluster=cluster1:1 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=local_cluster 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T01:15:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T01:15:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval local_cluster=cluster1 2024-04-24T01:15:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local_cluster=cluster1 2024-04-24T01:15:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T01:15:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T01:15:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T01:15:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T01:15:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-04-24T01:15:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-04-24T01:15:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-04-24T01:15:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local id_regexp= 2024-04-24T01:15:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:15:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:15:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:15:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:15:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:15:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:15:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:15:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:15:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-04-24T01:15:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local 'id_regexp=..*' 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-24T01:15:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-24T01:15:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local status_log=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-24T01:15:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ tee /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:minimum_set: 1 2024-04-24T01:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:active_set: 1 2024-04-24T01:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:registered clients: 2024-04-24T01:15:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: [id=4243e5f6-7584-4528-9e21-b1d37acf770f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6738], [object_number=7, tag_tid=1, entry_tid=6735]]], state=connected] 2024-04-24T01:15:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.status 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:15:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T01:15:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T01:15:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-04-24T01:15:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_50': 2024-04-24T01:15:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T01:15:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 122d6a7715c2 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.122d6a7715c2 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:04:59 2024 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:04:59 2024 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:04:59 2024 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 122d6a7715c2 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:15:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:15:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 3eb0b286-459e-4cd5-b044-62f0136dc9cd 2024-04-24T01:15:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T01:15:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_50 snap 2024-04-24T01:15:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:15:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T01:15:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T01:15:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.export 2024-04-24T01:15:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_50.export 2024-04-24T01:15:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_50.export 2024-04-24T01:15:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.export 2024-04-24T01:15:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:15:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_50.export 2024-04-24T01:15:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T01:15:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_50.export 2024-04-24T01:15:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.aEE7yWfHIo/cluster2-mirror-image_50.export /tmp/tmp.aEE7yWfHIo/cluster1-mirror-image_50.export 2024-04-24T01:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: delete many images' 2024-04-24T01:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.aEE7yWfHIo/rbd-mirror.test.log 2024-04-24T01:15:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T01:15:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 01:15:09 TEST: delete many images 2024-04-24T01:15:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 01:15:09 TEST: delete many images 2024-04-24T01:15:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 50 2024-04-24T01:15:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T01:15:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-04-24T01:15:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:15:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-04-24T01:15:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_1 2024-04-24T01:15:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:15:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_1 2024-04-24T01:15:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:15:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-04-24T01:15:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-04-24T01:15:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T01:15:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-04-24T01:15:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:15:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-04-24T01:15:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_2 2024-04-24T01:15:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:15:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_2 2024-04-24T01:15:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:15:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-04-24T01:15:11.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-04-24T01:15:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T01:15:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-04-24T01:15:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:15:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-04-24T01:15:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_3 2024-04-24T01:15:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:15:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_3 2024-04-24T01:15:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:15:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-04-24T01:15:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-04-24T01:15:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T01:15:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-04-24T01:15:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:15:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-04-24T01:15:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_4 2024-04-24T01:15:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:15:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_4 2024-04-24T01:15:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:15:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-04-24T01:15:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-04-24T01:15:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T01:15:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-04-24T01:15:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:15:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-04-24T01:15:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_5 2024-04-24T01:15:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:15:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_5 2024-04-24T01:15:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:15:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-04-24T01:15:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-04-24T01:15:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T01:15:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-04-24T01:15:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:15:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-04-24T01:15:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_6 2024-04-24T01:15:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:15:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_6 2024-04-24T01:15:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:15:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-04-24T01:15:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-04-24T01:15:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T01:15:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-04-24T01:15:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:15:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-04-24T01:15:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_7 2024-04-24T01:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_7 2024-04-24T01:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-04-24T01:15:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-04-24T01:15:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T01:15:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-04-24T01:15:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:15:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-04-24T01:15:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_8 2024-04-24T01:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_8 2024-04-24T01:15:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:15:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-04-24T01:15:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-04-24T01:15:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T01:15:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-04-24T01:15:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:15:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-04-24T01:15:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_9 2024-04-24T01:15:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:15:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_9 2024-04-24T01:15:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:15:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-04-24T01:15:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-04-24T01:15:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T01:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-04-24T01:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:15:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-04-24T01:15:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_10 2024-04-24T01:15:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:15:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_10 2024-04-24T01:15:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:15:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-04-24T01:15:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-04-24T01:15:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_11 2024-04-24T01:15:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-04-24T01:15:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:15:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-04-24T01:15:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_11 2024-04-24T01:15:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:15:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_11 2024-04-24T01:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-04-24T01:15:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-04-24T01:15:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_12 2024-04-24T01:15:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-04-24T01:15:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:15:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-04-24T01:15:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_12 2024-04-24T01:15:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:15:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_12 2024-04-24T01:15:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-04-24T01:15:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-04-24T01:15:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_13 2024-04-24T01:15:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-04-24T01:15:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:15:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-04-24T01:15:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_13 2024-04-24T01:15:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:15:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_13 2024-04-24T01:15:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:15:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-04-24T01:15:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-04-24T01:15:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_14 2024-04-24T01:15:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-04-24T01:15:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:15:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-04-24T01:15:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_14 2024-04-24T01:15:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:15:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_14 2024-04-24T01:15:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:15:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-04-24T01:15:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-04-24T01:15:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_15 2024-04-24T01:15:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-04-24T01:15:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:15:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-04-24T01:15:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_15 2024-04-24T01:15:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:15:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_15 2024-04-24T01:15:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:15:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-04-24T01:15:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-04-24T01:15:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_16 2024-04-24T01:15:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-04-24T01:15:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:15:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-04-24T01:15:24.503 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:15:24.504+0000 7fbdacd14640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2024-04-24T01:15:24.504 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:15:24.504+0000 7fbda950d640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2024-04-24T01:15:24.507 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:15:24.508+0000 7fbdadd16640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 11 2024-04-24T01:15:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_16 2024-04-24T01:15:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:15:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_16 2024-04-24T01:15:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:15:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-04-24T01:15:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-04-24T01:15:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_17 2024-04-24T01:15:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-04-24T01:15:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:15:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-04-24T01:15:25.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_17 2024-04-24T01:15:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:15:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_17 2024-04-24T01:15:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:15:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-04-24T01:15:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-04-24T01:15:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_18 2024-04-24T01:15:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-04-24T01:15:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:15:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-04-24T01:15:26.294 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:15:26.292+0000 7f6301759640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2024-04-24T01:15:26.295 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:15:26.292+0000 7f6307f66640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-04-24T01:15:26.295 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:15:26.292+0000 7f6307765640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 9 2024-04-24T01:15:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_18 2024-04-24T01:15:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:15:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_18 2024-04-24T01:15:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:15:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-04-24T01:15:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-04-24T01:15:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_19 2024-04-24T01:15:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-04-24T01:15:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:15:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-04-24T01:15:29.088 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2603193695517051) > weight (0.25) 2024-04-24T01:15:29.089 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6304396642831057) > weight (0.25) 2024-04-24T01:15:29.089 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.39488036986454944) > weight (0.25) 2024-04-24T01:15:29.089 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4668843081106713) > weight (0.25) 2024-04-24T01:15:29.089 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-04-24T01:15:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_19 2024-04-24T01:15:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:15:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_19 2024-04-24T01:15:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:15:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-04-24T01:15:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-04-24T01:15:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_20 2024-04-24T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-04-24T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-04-24T01:15:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_20 2024-04-24T01:15:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:15:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_20 2024-04-24T01:15:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:15:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-04-24T01:15:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-04-24T01:15:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_21 2024-04-24T01:15:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-04-24T01:15:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:15:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-04-24T01:15:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_21 2024-04-24T01:15:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:15:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_21 2024-04-24T01:15:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:15:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-04-24T01:15:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-04-24T01:15:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_22 2024-04-24T01:15:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-04-24T01:15:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:15:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-04-24T01:15:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_22 2024-04-24T01:15:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:15:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_22 2024-04-24T01:15:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:15:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-04-24T01:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-04-24T01:15:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_23 2024-04-24T01:15:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-04-24T01:15:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:15:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-04-24T01:15:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_23 2024-04-24T01:15:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:15:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_23 2024-04-24T01:15:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:15:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-04-24T01:15:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-04-24T01:15:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_24 2024-04-24T01:15:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-04-24T01:15:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:15:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-04-24T01:15:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_24 2024-04-24T01:15:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:15:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_24 2024-04-24T01:15:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:15:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-04-24T01:15:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-04-24T01:15:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_25 2024-04-24T01:15:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-04-24T01:15:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:15:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-04-24T01:15:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_25 2024-04-24T01:15:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:15:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_25 2024-04-24T01:15:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:15:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-04-24T01:15:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-04-24T01:15:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_26 2024-04-24T01:15:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-04-24T01:15:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:15:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-04-24T01:15:35.591 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:15:35.587+0000 7f6303f5e640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2024-04-24T01:15:35.593 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:15:35.591+0000 7f6306763640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 11 2024-04-24T01:15:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_26 2024-04-24T01:15:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:15:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_26 2024-04-24T01:15:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:15:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-04-24T01:15:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-04-24T01:15:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_27 2024-04-24T01:15:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-04-24T01:15:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:15:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-04-24T01:15:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_27 2024-04-24T01:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_27 2024-04-24T01:15:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:15:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-04-24T01:15:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-04-24T01:15:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_28 2024-04-24T01:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-04-24T01:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-04-24T01:15:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_28 2024-04-24T01:15:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:15:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_28 2024-04-24T01:15:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:15:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-04-24T01:15:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-04-24T01:15:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_29 2024-04-24T01:15:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-04-24T01:15:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:15:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-04-24T01:15:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_29 2024-04-24T01:15:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:15:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_29 2024-04-24T01:15:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:15:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-04-24T01:15:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-04-24T01:15:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_30 2024-04-24T01:15:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-04-24T01:15:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:15:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-04-24T01:15:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_30 2024-04-24T01:15:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:15:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_30 2024-04-24T01:15:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:15:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-04-24T01:15:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-04-24T01:15:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_31 2024-04-24T01:15:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-04-24T01:15:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:15:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-04-24T01:15:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_31 2024-04-24T01:15:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:15:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_31 2024-04-24T01:15:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:15:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-04-24T01:15:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-04-24T01:15:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_32 2024-04-24T01:15:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-04-24T01:15:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:15:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-04-24T01:15:41.307 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:15:41.303+0000 7fbdadd16640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 8 2024-04-24T01:15:41.307 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T01:15:41.307+0000 7fbdacd14640 -1 bdev(0x5622ce0e8e00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 11 2024-04-24T01:15:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_32 2024-04-24T01:15:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:15:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_32 2024-04-24T01:15:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:15:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-04-24T01:15:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-04-24T01:15:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_33 2024-04-24T01:15:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-04-24T01:15:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:15:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-04-24T01:15:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_33 2024-04-24T01:15:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:15:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_33 2024-04-24T01:15:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:15:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-04-24T01:15:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-04-24T01:15:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_34 2024-04-24T01:15:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-04-24T01:15:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:15:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-04-24T01:15:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_34 2024-04-24T01:15:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:15:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_34 2024-04-24T01:15:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:15:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-04-24T01:15:44.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-04-24T01:15:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_35 2024-04-24T01:15:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-04-24T01:15:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:15:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-04-24T01:15:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_35 2024-04-24T01:15:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:15:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_35 2024-04-24T01:15:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:15:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-04-24T01:15:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-04-24T01:15:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_36 2024-04-24T01:15:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-04-24T01:15:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:15:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-04-24T01:15:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_36 2024-04-24T01:15:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:15:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_36 2024-04-24T01:15:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:15:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-04-24T01:15:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-04-24T01:15:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_37 2024-04-24T01:15:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-04-24T01:15:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:15:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-04-24T01:15:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_37 2024-04-24T01:15:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:15:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_37 2024-04-24T01:15:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:15:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-04-24T01:15:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-04-24T01:15:47.384 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:15:47.385+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9d97860 2024-04-24T01:15:47.386 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:15:47.385+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dc9f08ea0 2024-04-24T01:15:47.389 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:15:47.385+0000 7fbc3ce8e640 -1 reset not still connected to 0x556dca53c0d0 2024-04-24T01:15:47.389 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:15:47.385+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de1885a00 2024-04-24T01:15:47.389 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:15:47.385+0000 7fbc3ce8e640 -1 reset not still connected to 0x556de2948d00 2024-04-24T01:15:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_38 2024-04-24T01:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-04-24T01:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:15:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-04-24T01:15:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_38 2024-04-24T01:15:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:15:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_38 2024-04-24T01:15:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:15:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-04-24T01:15:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-04-24T01:15:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_39 2024-04-24T01:15:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-04-24T01:15:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:15:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-04-24T01:15:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_39 2024-04-24T01:15:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:15:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_39 2024-04-24T01:15:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:15:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-04-24T01:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-04-24T01:15:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_40 2024-04-24T01:15:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-04-24T01:15:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:15:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-04-24T01:15:49.676 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:15:49.675+0000 7f6307f66640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 20 2024-04-24T01:15:49.677 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:15:49.675+0000 7f630375d640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 24 2024-04-24T01:15:49.678 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:15:49.675+0000 7f6301759640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 25 2024-04-24T01:15:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_40 2024-04-24T01:15:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:15:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_40 2024-04-24T01:15:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:15:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-04-24T01:15:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-04-24T01:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_41 2024-04-24T01:15:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-04-24T01:15:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:15:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-04-24T01:15:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_41 2024-04-24T01:15:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:15:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_41 2024-04-24T01:15:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:15:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-04-24T01:15:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-04-24T01:15:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_42 2024-04-24T01:15:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-04-24T01:15:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:15:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-04-24T01:15:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_42 2024-04-24T01:15:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:15:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_42 2024-04-24T01:15:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:15:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-04-24T01:15:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-04-24T01:15:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_43 2024-04-24T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-04-24T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-04-24T01:15:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_43 2024-04-24T01:15:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:15:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_43 2024-04-24T01:15:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:15:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-04-24T01:15:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-04-24T01:15:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_44 2024-04-24T01:15:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-04-24T01:15:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:15:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-04-24T01:15:53.693 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T01:15:53.691+0000 7f6307f66640 -1 bdev(0x55c523caee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2024-04-24T01:15:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_44 2024-04-24T01:15:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:15:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_44 2024-04-24T01:15:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:15:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-04-24T01:15:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-04-24T01:15:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_45 2024-04-24T01:15:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-04-24T01:15:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:15:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-04-24T01:15:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_45 2024-04-24T01:15:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:15:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_45 2024-04-24T01:15:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:15:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-04-24T01:15:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-04-24T01:15:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_46 2024-04-24T01:15:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-04-24T01:15:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:15:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-04-24T01:15:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_46 2024-04-24T01:15:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:15:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_46 2024-04-24T01:15:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:15:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-04-24T01:15:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-04-24T01:15:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_47 2024-04-24T01:15:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-04-24T01:15:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:15:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-04-24T01:15:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_47 2024-04-24T01:15:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:15:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_47 2024-04-24T01:15:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:15:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-04-24T01:15:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-04-24T01:15:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_48 2024-04-24T01:15:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-04-24T01:15:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:15:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-04-24T01:15:57.671 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:15:57.669+0000 7fd951b0e640 -1 reset not still connected to 0x563719a5af70 2024-04-24T01:15:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_48 2024-04-24T01:15:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:15:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_48 2024-04-24T01:15:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:15:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-04-24T01:15:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-04-24T01:15:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_49 2024-04-24T01:15:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-04-24T01:15:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:15:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-04-24T01:15:59.140 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:15:59.137+0000 7fd951b0e640 -1 reset not still connected to 0x5636fbe1e680 2024-04-24T01:15:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_49 2024-04-24T01:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_49 2024-04-24T01:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-04-24T01:15:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-04-24T01:15:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:15:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:15:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:15:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_50 2024-04-24T01:15:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-04-24T01:15:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:15:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:15:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:15:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T01:15:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-04-24T01:16:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T01:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_50 2024-04-24T01:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_50 2024-04-24T01:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-04-24T01:16:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-04-24T01:16:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T01:16:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: image deletions should propagate' 2024-04-24T01:16:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.aEE7yWfHIo/rbd-mirror.test.log 2024-04-24T01:16:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T01:16:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 01:16:00 TEST: image deletions should propagate 2024-04-24T01:16:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 01:16:00 TEST: image deletions should propagate 2024-04-24T01:16:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-04-24T01:16:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_count=0 2024-04-24T01:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local count 2024-04-24T01:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_count=0 2024-04-24T01:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T01:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T01:16:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T01:16:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T01:16:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-24T01:16:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'images: ' 2024-04-24T01:16:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 2 2024-04-24T01:16:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T01:16:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ count=0 2024-04-24T01:16:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T01:16:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-04-24T01:16:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state 2024-04-24T01:16:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T01:16:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T01:16:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T01:16:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-24T01:16:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'image health:' 2024-04-24T01:16:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 3 2024-04-24T01:16:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T01:16:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=OK 2024-04-24T01:16:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = ERROR 2024-04-24T01:16:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = OK 2024-04-24T01:16:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 50 2024-04-24T01:16:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:16:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T01:16:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-24T01:16:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-04-24T01:16:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T01:16:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:16:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T01:16:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-24T01:16:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T01:16:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-24T01:16:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-04-24T01:16:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:16:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T01:16:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:16:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T01:16:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-24T01:16:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T01:16:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:16:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T01:16:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-24T01:16:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-04-24T01:16:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:16:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T01:16:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:16:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T01:16:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-24T01:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:16:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T01:16:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-24T01:16:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-04-24T01:16:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:16:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T01:16:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:16:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T01:16:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-24T01:16:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T01:16:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:16:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T01:16:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-24T01:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-04-24T01:16:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:16:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T01:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T01:16:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-24T01:16:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:16:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T01:16:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-24T01:16:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-04-24T01:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T01:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:16:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T01:16:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T01:16:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:16:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T01:16:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-24T01:16:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-04-24T01:16:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:16:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T01:16:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:16:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T01:16:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T01:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T01:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-24T01:16:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-04-24T01:16:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:16:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T01:16:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:16:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T01:16:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-24T01:16:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T01:16:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:16:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T01:16:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-24T01:16:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-04-24T01:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T01:16:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:16:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T01:16:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T01:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T01:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-24T01:16:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-04-24T01:16:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:16:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_11 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:16:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_11 2024-04-24T01:16:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:16:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-04-24T01:16:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-04-24T01:16:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-04-24T01:16:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T01:16:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_11 2024-04-24T01:16:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T01:16:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-04-24T01:16:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-04-24T01:16:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_12 2024-04-24T01:16:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-04-24T01:16:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_12 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:16:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-04-24T01:16:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-04-24T01:16:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-04-24T01:16:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T01:16:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_12 2024-04-24T01:16:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T01:16:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-04-24T01:16:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-04-24T01:16:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_13 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_13 2024-04-24T01:16:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_13 2024-04-24T01:16:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-04-24T01:16:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-04-24T01:16:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-04-24T01:16:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T01:16:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_13 2024-04-24T01:16:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_13 2024-04-24T01:16:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-04-24T01:16:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-04-24T01:16:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_14 2024-04-24T01:16:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-04-24T01:16:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:16:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_14 2024-04-24T01:16:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:16:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-04-24T01:16:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-04-24T01:16:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_14 2024-04-24T01:16:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T01:16:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-04-24T01:16:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-04-24T01:16:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_15 2024-04-24T01:16:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-04-24T01:16:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_15 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:16:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-04-24T01:16:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-04-24T01:16:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-04-24T01:16:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T01:16:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_15 2024-04-24T01:16:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T01:16:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-04-24T01:16:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-04-24T01:16:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_16 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_16 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:16:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-04-24T01:16:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-04-24T01:16:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-04-24T01:16:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T01:16:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_16 2024-04-24T01:16:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T01:16:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-04-24T01:16:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-04-24T01:16:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_17 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_17 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:16:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-04-24T01:16:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-04-24T01:16:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-04-24T01:16:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T01:16:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_17 2024-04-24T01:16:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T01:16:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-04-24T01:16:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-04-24T01:16:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_18 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_18 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_18 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-04-24T01:16:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-04-24T01:16:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-04-24T01:16:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T01:16:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_18 2024-04-24T01:16:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_18 2024-04-24T01:16:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-04-24T01:16:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-04-24T01:16:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_19 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_19 2024-04-24T01:16:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:16:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-04-24T01:16:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-04-24T01:16:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-04-24T01:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T01:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_19 2024-04-24T01:16:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T01:16:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-04-24T01:16:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_20 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_20 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-04-24T01:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-04-24T01:16:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-04-24T01:16:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T01:16:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_20 2024-04-24T01:16:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T01:16:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-04-24T01:16:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-04-24T01:16:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_21 2024-04-24T01:16:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-04-24T01:16:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:16:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_21 2024-04-24T01:16:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:16:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-04-24T01:16:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-04-24T01:16:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_21 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T01:16:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-04-24T01:16:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_22 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:16:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_22 2024-04-24T01:16:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:16:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-04-24T01:16:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-04-24T01:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-04-24T01:16:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T01:16:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_22 2024-04-24T01:16:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T01:16:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-04-24T01:16:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-04-24T01:16:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_23 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_23 2024-04-24T01:16:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:16:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-04-24T01:16:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-04-24T01:16:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-04-24T01:16:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T01:16:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_23 2024-04-24T01:16:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T01:16:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-04-24T01:16:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-04-24T01:16:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_24 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_24 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_24 2024-04-24T01:16:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-04-24T01:16:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-04-24T01:16:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-04-24T01:16:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T01:16:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_24 2024-04-24T01:16:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_24 2024-04-24T01:16:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-04-24T01:16:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-04-24T01:16:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_25 2024-04-24T01:16:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-04-24T01:16:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_25 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-04-24T01:16:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-04-24T01:16:05.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-04-24T01:16:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T01:16:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_25 2024-04-24T01:16:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T01:16:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-04-24T01:16:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-04-24T01:16:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_26 2024-04-24T01:16:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-04-24T01:16:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_26 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-04-24T01:16:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-04-24T01:16:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_26 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T01:16:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-04-24T01:16:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-04-24T01:16:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_27 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_27 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:16:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-04-24T01:16:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-04-24T01:16:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_27 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T01:16:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-04-24T01:16:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-04-24T01:16:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_28 2024-04-24T01:16:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-04-24T01:16:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_28 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_28 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-04-24T01:16:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-04-24T01:16:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-04-24T01:16:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T01:16:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_28 2024-04-24T01:16:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_28 2024-04-24T01:16:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-04-24T01:16:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-04-24T01:16:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_29 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_29 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_29 2024-04-24T01:16:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-04-24T01:16:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-04-24T01:16:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-04-24T01:16:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T01:16:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_29 2024-04-24T01:16:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_29 2024-04-24T01:16:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-04-24T01:16:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-04-24T01:16:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_30 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_30 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:16:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-04-24T01:16:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-04-24T01:16:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-04-24T01:16:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T01:16:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_30 2024-04-24T01:16:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T01:16:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-04-24T01:16:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_31 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_31 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-04-24T01:16:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-04-24T01:16:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-04-24T01:16:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T01:16:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_31 2024-04-24T01:16:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T01:16:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-04-24T01:16:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-04-24T01:16:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_32 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_32 2024-04-24T01:16:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:16:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-04-24T01:16:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-04-24T01:16:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-04-24T01:16:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T01:16:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_32 2024-04-24T01:16:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T01:16:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-04-24T01:16:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_33 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_33 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-04-24T01:16:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-04-24T01:16:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-04-24T01:16:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T01:16:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_33 2024-04-24T01:16:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T01:16:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-04-24T01:16:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_34 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_34 2024-04-24T01:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-04-24T01:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-04-24T01:16:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-04-24T01:16:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T01:16:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_34 2024-04-24T01:16:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T01:16:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-04-24T01:16:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-04-24T01:16:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_35 2024-04-24T01:16:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-04-24T01:16:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_35 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-04-24T01:16:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-04-24T01:16:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-04-24T01:16:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T01:16:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_35 2024-04-24T01:16:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T01:16:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-04-24T01:16:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-04-24T01:16:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_36 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_36 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-04-24T01:16:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-04-24T01:16:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-04-24T01:16:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T01:16:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_36 2024-04-24T01:16:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T01:16:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-04-24T01:16:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-04-24T01:16:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_37 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_37 2024-04-24T01:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-04-24T01:16:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-04-24T01:16:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-04-24T01:16:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T01:16:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_37 2024-04-24T01:16:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T01:16:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-04-24T01:16:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-04-24T01:16:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_38 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_38 2024-04-24T01:16:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:16:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-04-24T01:16:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-04-24T01:16:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-04-24T01:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T01:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_38 2024-04-24T01:16:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T01:16:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-04-24T01:16:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-04-24T01:16:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_39 2024-04-24T01:16:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_39 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_39 2024-04-24T01:16:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-04-24T01:16:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-04-24T01:16:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-04-24T01:16:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T01:16:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_39 2024-04-24T01:16:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_39 2024-04-24T01:16:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-04-24T01:16:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_40 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_40 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-04-24T01:16:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-04-24T01:16:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-04-24T01:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T01:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_40 2024-04-24T01:16:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T01:16:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-04-24T01:16:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_41 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_41 2024-04-24T01:16:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:16:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-04-24T01:16:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-04-24T01:16:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-04-24T01:16:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T01:16:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_41 2024-04-24T01:16:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T01:16:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-04-24T01:16:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-04-24T01:16:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_42 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_42 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:16:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-04-24T01:16:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-04-24T01:16:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-04-24T01:16:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T01:16:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_42 2024-04-24T01:16:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T01:16:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-04-24T01:16:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-04-24T01:16:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_43 2024-04-24T01:16:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_43 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-04-24T01:16:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-04-24T01:16:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_43 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T01:16:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-04-24T01:16:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-04-24T01:16:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_44 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_44 2024-04-24T01:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-04-24T01:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-04-24T01:16:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-04-24T01:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T01:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_44 2024-04-24T01:16:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T01:16:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-04-24T01:16:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-04-24T01:16:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_45 2024-04-24T01:16:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_45 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-04-24T01:16:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-04-24T01:16:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-04-24T01:16:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_45 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T01:16:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-04-24T01:16:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-04-24T01:16:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_46 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_46 2024-04-24T01:16:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:16:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-04-24T01:16:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-04-24T01:16:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-04-24T01:16:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_46 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T01:16:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-04-24T01:16:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_47 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_47 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-04-24T01:16:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-04-24T01:16:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-04-24T01:16:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T01:16:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_47 2024-04-24T01:16:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T01:16:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-04-24T01:16:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-04-24T01:16:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_48 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_48 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:16:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-04-24T01:16:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-04-24T01:16:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-04-24T01:16:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T01:16:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_48 2024-04-24T01:16:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T01:16:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-04-24T01:16:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-04-24T01:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_49 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_49 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:16:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-04-24T01:16:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-04-24T01:16:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-04-24T01:16:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T01:16:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_49 2024-04-24T01:16:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T01:16:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-04-24T01:16:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-04-24T01:16:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_50 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_50 2024-04-24T01:16:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:16:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-04-24T01:16:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-04-24T01:16:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-04-24T01:16:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T01:16:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_50 2024-04-24T01:16:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T01:16:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-04-24T01:16:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-04-24T01:16:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.aEE7yWfHIo/rbd-mirror.test.log 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 01:16:10 TEST: delete images during bootstrap 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 01:16:10 TEST: delete images during bootstrap 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=image 2024-04-24T01:16:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-04-24T01:16:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:note: changing mirroring mode from pool to image 2024-04-24T01:16:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-04-24T01:16:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=image 2024-04-24T01:16:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-04-24T01:16:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:note: changing mirroring mode from pool to image 2024-04-24T01:16:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1 cluster instance 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=cluster1 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 = cluster1 ']' 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T01:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T01:16:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T01:16:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T01:16:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T01:16:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T01:16:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T01:16:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=test 2024-04-24T01:16:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 10 2024-04-24T01:16:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:16:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_1 512M 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-04-24T01:16:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_1 2024-04-24T01:16:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:16:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 journal 2024-04-24T01:16:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:16:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-04-24T01:16:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_1': 2024-04-24T01:16:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:16:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:16:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 35d6f879c12c 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.35d6f879c12c 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:16:11 2024 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:16:11 2024 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:16:11 2024 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 35d6f879c12c 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 13077dce-d633-4474-89d2-4a8e80991b42 2024-04-24T01:16:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:16:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_1 2024-04-24T01:16:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:16:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T01:16:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:16:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:16:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 34272256 bytes 2024-04-24T01:16:16.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 108809216 bytes 2024-04-24T01:16:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 170627072 bytes 2024-04-24T01:16:23.170 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:16:23.166+0000 7ff578350640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2024-04-24T01:16:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:16:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:16:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:16:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-04-24T01:16:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:16:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T01:16:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:16:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T01:16:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d83acc838a 2024-04-24T01:16:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_1 present 10d83acc838a 2024-04-24T01:16:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:16:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d83acc838a 2024-04-24T01:16:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T01:16:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:16:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T01:16:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d83acc838a 2024-04-24T01:16:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d83acc838a 2024-04-24T01:16:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d83acc838a -o 10d83acc838a = 10d83acc838a 2024-04-24T01:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_1 2024-04-24T01:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-04-24T01:16:29.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.35327967201891597) > weight (0.25) 2024-04-24T01:16:29.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.46720274044649746) > weight (0.25) 2024-04-24T01:16:29.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5256372929118738) > weight (0.25) 2024-04-24T01:16:29.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7934014602761923) > weight (0.25) 2024-04-24T01:16:29.595 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-04-24T01:16:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:16:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-04-24T01:16:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:16:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:16:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T01:16:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:16:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T01:16:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-24T01:16:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:16:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:16:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:16:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-04-24T01:16:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:16:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:16:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:16:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:16:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:16:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T01:16:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:16:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:16:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T01:16:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T01:16:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:16:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-24T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_1 2024-04-24T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-04-24T01:16:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_1 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_1 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T01:16:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-04-24T01:16:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-04-24T01:16:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_2 512M 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:16:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:16:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:16:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:16:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:16:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:16:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-04-24T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_2 2024-04-24T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 journal 2024-04-24T01:16:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:16:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-04-24T01:16:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_2': 2024-04-24T01:16:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:16:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:16:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:16:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 35ee72550013 2024-04-24T01:16:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.35ee72550013 2024-04-24T01:16:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:16:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:16:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:16:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:16:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:16:56 2024 2024-04-24T01:16:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:16:56 2024 2024-04-24T01:16:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:16:56 2024 2024-04-24T01:16:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 35ee72550013 2024-04-24T01:16:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:16:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:16:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e7cb16ee-a891-459b-a230-a674dae355b1 2024-04-24T01:16:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:16:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_2 2024-04-24T01:16:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:16:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:16:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:16:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:16:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=8 2024-04-24T01:16:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:16:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:17:01.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 58354688 bytes 2024-04-24T01:17:03.201 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:17:03.197+0000 7ff57634c640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 8 2024-04-24T01:17:03.203 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T01:17:03.201+0000 7ff576b4d640 -1 bdev(0x5613d52c4e00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 9 2024-04-24T01:17:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 82768896 bytes 2024-04-24T01:17:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:17:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T01:17:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:17:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:17:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:17:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T01:17:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:17:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d82057dd79 2024-04-24T01:17:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:17:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:17:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_2 present 10d82057dd79 2024-04-24T01:17:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:17:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:17:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:17:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d82057dd79 2024-04-24T01:17:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:17:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:17:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T01:17:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:17:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:17:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:17:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T01:17:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:17:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d82057dd79 2024-04-24T01:17:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d82057dd79 2024-04-24T01:17:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d82057dd79 -o 10d82057dd79 = 10d82057dd79 2024-04-24T01:17:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:17:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:17:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:17:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_2 2024-04-24T01:17:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:17:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:17:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-04-24T01:17:13.090 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6428094151717519) > weight (0.25) 2024-04-24T01:17:13.090 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9797254867147387) > weight (0.25) 2024-04-24T01:17:13.091 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.30524883204876974) > weight (0.25) 2024-04-24T01:17:13.091 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9232593757954499) > weight (0.25) 2024-04-24T01:17:13.091 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-04-24T01:17:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:17:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:17:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:17:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T01:17:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:17:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-24T01:17:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:17:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:17:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:17:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-04-24T01:17:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:17:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T01:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T01:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T01:17:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:17:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-24T01:17:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:17:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:17:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:17:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:17:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_2 2024-04-24T01:17:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:17:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:17:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-04-24T01:17:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_2 2024-04-24T01:17:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:17:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:17:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:17:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:17:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_2 2024-04-24T01:17:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:17:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T01:17:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-04-24T01:17:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-04-24T01:17:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:17:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:17:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:17:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T01:17:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_3 512M 2024-04-24T01:17:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:17:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:17:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:17:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:17:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:17:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:17:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:17:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:17:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:17:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-04-24T01:17:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_3 2024-04-24T01:17:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:17:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:17:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:17:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 journal 2024-04-24T01:17:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:17:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-04-24T01:17:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_3': 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 360636b6082b 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.360636b6082b 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:17:33 2024 2024-04-24T01:17:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:17:33 2024 2024-04-24T01:17:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:17:33 2024 2024-04-24T01:17:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 360636b6082b 2024-04-24T01:17:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:17:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:17:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 1488692d-49a0-48ab-b4ed-a94c9585070b 2024-04-24T01:17:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:17:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_3 2024-04-24T01:17:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:17:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:17:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:17:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T01:17:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:17:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:17:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 77799424 bytes 2024-04-24T01:17:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 131777536 bytes 2024-04-24T01:17:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:17:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:17:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:17:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T01:17:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:17:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d82ac220c8 2024-04-24T01:17:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:17:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:17:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_3 present 10d82ac220c8 2024-04-24T01:17:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:17:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:17:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:17:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d82ac220c8 2024-04-24T01:17:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:17:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:17:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T01:17:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:17:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:17:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:17:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T01:17:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:17:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d82ac220c8 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d82ac220c8 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d82ac220c8 -o 10d82ac220c8 = 10d82ac220c8 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_3 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:17:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-04-24T01:17:44.289 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-04-24T01:17:44.290+0000 7fd951b0e640 -1 reset not still connected to 0x563719a5af70 2024-04-24T01:18:05.596 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6272260578596519) > weight (0.25) 2024-04-24T01:18:05.597 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.43550627719731594) > weight (0.25) 2024-04-24T01:18:05.597 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6637547537703536) > weight (0.25) 2024-04-24T01:18:05.597 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.41310497816414893) > weight (0.25) 2024-04-24T01:18:05.597 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-24T01:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:18:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-04-24T01:18:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:18:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:18:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:18:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:18:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:18:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:18:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T01:18:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:18:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:18:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:18:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T01:18:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:18:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-24T01:18:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:18:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:18:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:18:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-04-24T01:18:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:18:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:18:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:18:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:18:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:18:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:18:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T01:18:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:18:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:18:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T01:18:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T01:18:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:18:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-24T01:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:18:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:18:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_3 2024-04-24T01:18:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:18:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-04-24T01:18:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_3 2024-04-24T01:18:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:18:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:18:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:18:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_3 2024-04-24T01:18:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T01:18:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-04-24T01:18:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-04-24T01:18:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:18:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:18:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:18:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T01:18:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_4 512M 2024-04-24T01:18:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:18:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:18:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:18:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:18:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:18:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:18:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:18:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-04-24T01:18:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_4 2024-04-24T01:18:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:18:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 journal 2024-04-24T01:18:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:18:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-04-24T01:18:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_4': 2024-04-24T01:18:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:18:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:18:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 361e6f0b388f 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.361e6f0b388f 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:18:09 2024 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:18:09 2024 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:18:09 2024 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 361e6f0b388f 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:18:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e91f3158-ae6b-46f2-8e10-1d410b865431 2024-04-24T01:18:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_4 2024-04-24T01:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:18:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T01:18:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:18:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:18:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 27992064 bytes 2024-04-24T01:18:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 76689408 bytes 2024-04-24T01:18:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 138499072 bytes 2024-04-24T01:18:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:18:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:18:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:18:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:18:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-04-24T01:18:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:18:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T01:18:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d8e26ce42f 2024-04-24T01:18:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:18:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:18:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_4 present 10d8e26ce42f 2024-04-24T01:18:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d8e26ce42f 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:18:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:18:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T01:18:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:18:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d8e26ce42f 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d8e26ce42f 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d8e26ce42f -o 10d8e26ce42f = 10d8e26ce42f 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_4 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-04-24T01:18:23.123 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-04-24T01:18:23.122+0000 7fbc3ce8e640 -1 reset not still connected to 0x556daaeddc70 2024-04-24T01:18:33.092 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.822215800051277) > weight (0.25) 2024-04-24T01:18:33.092 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.968476607372171) > weight (0.25) 2024-04-24T01:18:33.092 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.38939715867611546) > weight (0.25) 2024-04-24T01:18:33.092 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9997812561857885) > weight (0.25) 2024-04-24T01:18:33.092 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-04-24T01:18:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:18:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-04-24T01:18:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T01:18:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:18:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-24T01:18:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:18:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:18:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:18:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-04-24T01:18:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:18:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:18:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:18:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:18:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:18:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T01:18:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:18:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:18:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T01:18:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T01:18:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:18:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-24T01:18:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:18:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:18:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:18:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:18:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_4 2024-04-24T01:18:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-04-24T01:18:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_4 2024-04-24T01:18:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:18:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:18:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_4 2024-04-24T01:18:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T01:18:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-04-24T01:18:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-04-24T01:18:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_5 512M 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:18:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:18:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:18:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:18:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:18:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:18:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-04-24T01:18:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_5 2024-04-24T01:18:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:18:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:18:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 journal 2024-04-24T01:18:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:18:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_5': 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 3636ea038718 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.3636ea038718 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:18:53 2024 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:18:53 2024 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:18:53 2024 2024-04-24T01:18:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 3636ea038718 2024-04-24T01:18:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:18:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:18:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 073ac9aa-a07a-49e6-b941-eff1456441e8 2024-04-24T01:18:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:18:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_5 2024-04-24T01:18:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:18:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:18:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:18:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:18:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T01:18:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:18:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:18:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 50495488 bytes 2024-04-24T01:19:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 104996864 bytes 2024-04-24T01:19:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 226377728 bytes 2024-04-24T01:19:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:19:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:19:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:19:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:19:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T01:19:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:19:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d875662c0b 2024-04-24T01:19:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:19:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:19:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_5 present 10d875662c0b 2024-04-24T01:19:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:19:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:19:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:19:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d875662c0b 2024-04-24T01:19:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:19:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:19:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T01:19:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:19:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:19:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:19:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T01:19:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:19:07.597 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T01:19:07.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T01:19:07.598 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-24T01:19:07.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:19:07.599+0000 7f727e86b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 98241) UID: 1000 2024-04-24T01:19:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d875662c0b 2024-04-24T01:19:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d875662c0b 2024-04-24T01:19:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d875662c0b -o 10d875662c0b = 10d875662c0b 2024-04-24T01:19:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:19:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:19:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:19:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_5 2024-04-24T01:19:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:19:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:19:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-04-24T01:19:11.599 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T01:19:11.599 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:19:11.599 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T01:19:11.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T01:19:11.600 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T01:19:11.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T01:19:11.603 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-04-24T01:19:12.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:19:12.611+0000 7f0a8ac66640 -1 rbd::mirror::ImageReplayer: 0x557e03cbe500 [3/073ac9aa-a07a-49e6-b941-eff1456441e8] start: already running 2024-04-24T01:19:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:19:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-04-24T01:19:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:19:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:19:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:19:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:19:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:19:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:19:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T01:19:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:19:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:19:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:19:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T01:19:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:19:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-24T01:19:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:19:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:19:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:19:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-04-24T01:19:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:19:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:19:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:19:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:19:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:19:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:19:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T01:19:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:19:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:19:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T01:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T01:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:19:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-24T01:19:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:19:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:19:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:19:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:19:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_5 2024-04-24T01:19:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:19:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:19:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-04-24T01:19:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_5 2024-04-24T01:19:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:19:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:19:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:19:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:19:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_5 2024-04-24T01:19:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:19:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T01:19:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-04-24T01:19:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-04-24T01:19:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:19:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:19:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:19:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T01:19:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_6 512M 2024-04-24T01:19:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:19:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-04-24T01:19:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_6 2024-04-24T01:19:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:19:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:19:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:19:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 journal 2024-04-24T01:19:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:19:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-04-24T01:19:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_6': 2024-04-24T01:19:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 365330c4d9ee 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.365330c4d9ee 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:19:36 2024 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:19:36 2024 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:19:36 2024 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 365330c4d9ee 2024-04-24T01:19:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:19:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:19:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 466203e5-3433-47b6-9fbf-bd439a2b18f6 2024-04-24T01:19:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:19:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_6 2024-04-24T01:19:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:19:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:19:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:19:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T01:19:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:19:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:19:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 28150784 bytes 2024-04-24T01:19:40.093 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T01:19:40.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T01:19:40.094 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-24T01:19:40.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:19:40.094+0000 7f300d023640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 111256) UID: 1000 2024-04-24T01:19:41.095 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T01:19:41.095 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:19:41.096 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T01:19:41.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T01:19:41.096 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T01:19:41.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T01:19:41.098 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-04-24T01:19:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 82996224 bytes 2024-04-24T01:19:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 119737344 bytes 2024-04-24T01:19:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 141262848 bytes 2024-04-24T01:19:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 141895680 bytes 2024-04-24T01:19:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:19:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:19:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:19:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:19:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-04-24T01:19:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:19:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:19:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:19:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:19:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:19:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:19:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T01:19:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:19:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:19:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:19:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:19:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T01:19:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d8f803d86 2024-04-24T01:19:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:19:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:19:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_6 present 10d8f803d86 2024-04-24T01:19:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:19:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:19:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:19:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d8f803d86 2024-04-24T01:19:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:19:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:19:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T01:19:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:19:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:19:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:19:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T01:19:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:19:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d8f803d86 2024-04-24T01:19:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d8f803d86 2024-04-24T01:19:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d8f803d86 -o 10d8f803d86 = 10d8f803d86 2024-04-24T01:19:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:19:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:19:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:19:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_6 2024-04-24T01:19:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:19:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:19:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:19:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-04-24T01:20:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:20:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:20:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T01:20:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:20:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-24T01:20:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:20:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:20:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:20:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-04-24T01:20:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:20:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:20:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:20:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:20:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:20:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:20:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T01:20:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:20:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:20:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T01:20:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T01:20:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-24T01:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_6 2024-04-24T01:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-04-24T01:20:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_6 2024-04-24T01:20:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:20:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:20:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:20:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:20:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_6 2024-04-24T01:20:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:20:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T01:20:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-04-24T01:20:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-04-24T01:20:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:20:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:20:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:20:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_7 512M 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:20:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-04-24T01:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_7 2024-04-24T01:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 journal 2024-04-24T01:20:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:20:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_7': 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 367378d4c502 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.367378d4c502 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:20:17 2024 2024-04-24T01:20:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:20:17 2024 2024-04-24T01:20:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:20:17 2024 2024-04-24T01:20:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 367378d4c502 2024-04-24T01:20:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:20:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:20:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: eef38333-9878-42b8-8569-f5a35e3ce676 2024-04-24T01:20:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:20:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_7 2024-04-24T01:20:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:20:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:20:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:20:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T01:20:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:20:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:20:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 47877120 bytes 2024-04-24T01:20:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 48524288 bytes 2024-04-24T01:20:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 98521088 bytes 2024-04-24T01:20:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 131124224 bytes 2024-04-24T01:20:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 155844608 bytes 2024-04-24T01:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 186084352 bytes 2024-04-24T01:20:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 197274624 bytes 2024-04-24T01:20:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T01:20:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:20:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:20:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:20:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:20:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T01:20:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d8529729ab 2024-04-24T01:20:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:20:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:20:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_7 present 10d8529729ab 2024-04-24T01:20:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:20:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:20:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:20:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d8529729ab 2024-04-24T01:20:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:20:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:20:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T01:20:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:20:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:20:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:20:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T01:20:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:20:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d8529729ab 2024-04-24T01:20:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d8529729ab 2024-04-24T01:20:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d8529729ab -o 10d8529729ab = 10d8529729ab 2024-04-24T01:20:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:20:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:20:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:20:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_7 2024-04-24T01:20:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:20:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:20:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:20:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-04-24T01:20:56.099 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6213006871150694) > weight (0.25) 2024-04-24T01:20:56.100 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T01:20:56.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T01:20:56.100 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-24T01:20:56.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:20:56.101+0000 7fd2f07ef640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 121037) UID: 1000 2024-04-24T01:21:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:21:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-04-24T01:21:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:21:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:21:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:21:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:21:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:21:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:21:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T01:21:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:21:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:21:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:21:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T01:21:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:21:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-24T01:21:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:21:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:21:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:21:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:21:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T01:21:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T01:21:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:21:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-24T01:21:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:21:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:21:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:21:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:21:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_7 2024-04-24T01:21:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:21:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-04-24T01:21:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_7 2024-04-24T01:21:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:21:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:21:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:21:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_7 2024-04-24T01:21:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T01:21:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-04-24T01:21:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-04-24T01:21:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_8 512M 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:21:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:21:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:21:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:21:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-04-24T01:21:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_8 2024-04-24T01:21:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:21:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 journal 2024-04-24T01:21:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:21:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-04-24T01:21:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_8': 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 368b6eb15e6b 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.368b6eb15e6b 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:21:02 2024 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:21:02 2024 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:21:02 2024 2024-04-24T01:21:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 368b6eb15e6b 2024-04-24T01:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 1b99de17-b9ad-430d-b935-a740387298b8 2024-04-24T01:21:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:21:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_8 2024-04-24T01:21:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:21:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T01:21:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:21:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:21:05.101 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T01:21:05.101 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T01:21:05.102 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T01:21:05.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T01:21:05.102 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T01:21:05.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T01:21:05.105 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-04-24T01:21:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 70542336 bytes 2024-04-24T01:21:10.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.48681376280495636) > weight (0.25) 2024-04-24T01:21:10.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.36503060885294003) > weight (0.25) 2024-04-24T01:21:10.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.34478236465234957) > weight (0.25) 2024-04-24T01:21:10.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.38058108243407407) > weight (0.25) 2024-04-24T01:21:10.604 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-04-24T01:21:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 129711104 bytes 2024-04-24T01:21:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 150921216 bytes 2024-04-24T01:21:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 175832064 bytes 2024-04-24T01:21:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:21:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:21:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:21:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T01:21:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:21:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d8f910d5f0 2024-04-24T01:21:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:21:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:21:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_8 present 10d8f910d5f0 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d8f910d5f0 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T01:21:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d8f910d5f0 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d8f910d5f0 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d8f910d5f0 -o 10d8f910d5f0 = 10d8f910d5f0 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_8 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-04-24T01:21:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:21:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-04-24T01:21:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:21:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T01:21:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:21:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-24T01:21:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:21:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:21:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:21:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-04-24T01:21:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:21:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:21:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:21:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:21:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:21:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T01:21:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:21:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:21:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T01:21:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T01:21:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:21:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-24T01:21:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:21:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:21:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:21:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:21:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_8 2024-04-24T01:21:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-04-24T01:21:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_8 2024-04-24T01:21:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:21:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:21:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_8 2024-04-24T01:21:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T01:21:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-04-24T01:21:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-04-24T01:21:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_9 512M 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-04-24T01:21:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_9 2024-04-24T01:21:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:21:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:21:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 journal 2024-04-24T01:21:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:21:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-04-24T01:21:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_9': 2024-04-24T01:21:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 36ad346d2c21 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.36ad346d2c21 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:21:45 2024 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:21:45 2024 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:21:45 2024 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 36ad346d2c21 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:21:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 296acce2-2930-421a-a136-cb2180ed9579 2024-04-24T01:21:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:21:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_9 2024-04-24T01:21:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:21:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:21:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=8 2024-04-24T01:21:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:21:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:21:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 43375616 bytes 2024-04-24T01:21:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 72448000 bytes 2024-04-24T01:21:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:21:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:21:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:21:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:21:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T01:21:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d8faf1c46a 2024-04-24T01:21:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:21:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:21:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_9 present 10d8faf1c46a 2024-04-24T01:21:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:21:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:21:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:21:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d8faf1c46a 2024-04-24T01:21:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:21:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:21:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T01:21:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:21:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:21:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:21:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T01:21:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:21:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d8faf1c46a 2024-04-24T01:21:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d8faf1c46a 2024-04-24T01:21:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d8faf1c46a -o 10d8faf1c46a = 10d8faf1c46a 2024-04-24T01:21:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:21:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:21:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:21:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_9 2024-04-24T01:21:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:21:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:21:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:21:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-04-24T01:22:09.106 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4269092909839779) > weight (0.25) 2024-04-24T01:22:09.106 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9769838590504786) > weight (0.25) 2024-04-24T01:22:09.106 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7990051368706299) > weight (0.25) 2024-04-24T01:22:09.106 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5656538700135715) > weight (0.25) 2024-04-24T01:22:09.106 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-04-24T01:22:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:22:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-04-24T01:22:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:22:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:22:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:22:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:22:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:22:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:22:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T01:22:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:22:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:22:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:22:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T01:22:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:22:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-24T01:22:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:22:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:22:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:22:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-04-24T01:22:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:22:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:22:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:22:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:22:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:22:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:22:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T01:22:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:22:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:22:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T01:22:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T01:22:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-24T01:22:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_9 2024-04-24T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-04-24T01:22:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_9 2024-04-24T01:22:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:22:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:22:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:22:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:22:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_9 2024-04-24T01:22:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:22:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T01:22:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-04-24T01:22:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-04-24T01:22:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:22:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:22:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T01:22:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T01:22:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_10 512M 2024-04-24T01:22:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:22:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:22:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:22:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:22:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:22:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T01:22:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T01:22:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T01:22:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T01:22:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-04-24T01:22:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_10 2024-04-24T01:22:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:22:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:22:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=journal 2024-04-24T01:22:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 journal 2024-04-24T01:22:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T01:22:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-04-24T01:22:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_10': 2024-04-24T01:22:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T01:22:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T01:22:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 0 2024-04-24T01:22:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 36c5f003c154 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.36c5f003c154 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, journaling 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 01:22:24 2024 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 01:22:24 2024 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 01:22:24 2024 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: journal: 36c5f003c154 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: journal 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 1ea45b72-2ec6-41af-ae08-7f1150b3a147 2024-04-24T01:22:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T01:22:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_10 2024-04-24T01:22:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:22:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:22:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T01:22:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T01:22:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:22:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-04-24T01:22:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 23633920 bytes 2024-04-24T01:22:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 46232576 bytes 2024-04-24T01:22:31.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:22:31.947+0000 7fb15a556640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5578d7483380 finish: resending after timeout 2024-04-24T01:22:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 99748864 bytes 2024-04-24T01:22:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 128246784 bytes 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:22:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:22:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:22:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T01:22:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:22:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:22:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:22:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T01:22:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:22:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10d8a33f12f9 2024-04-24T01:22:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:22:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:22:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_10 present 10d8a33f12f9 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10d8a33f12f9 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T01:22:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:22:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10d8a33f12f9 2024-04-24T01:22:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10d8a33f12f9 2024-04-24T01:22:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10d8a33f12f9 -o 10d8a33f12f9 = 10d8a33f12f9 2024-04-24T01:22:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T01:22:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T01:22:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:22:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_10 2024-04-24T01:22:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:22:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:22:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:22:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-04-24T01:22:39.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3377666721901257) > weight (0.25) 2024-04-24T01:22:39.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.542974446584024) > weight (0.25) 2024-04-24T01:22:39.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6448800210717588) > weight (0.25) 2024-04-24T01:22:39.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.57354737232602) > weight (0.25) 2024-04-24T01:22:39.606 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-04-24T01:23:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T01:23:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-04-24T01:23:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:23:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:23:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T01:23:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:23:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T01:23:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:23:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T01:23:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:23:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:23:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T01:23:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:23:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-24T01:23:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T01:23:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T01:23:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T01:23:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-04-24T01:23:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:23:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:23:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T01:23:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T01:23:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T01:23:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T01:23:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T01:23:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:23:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T01:23:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T01:23:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T01:23:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-24T01:23:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T01:23:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T01:23:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T01:23:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:23:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_10 2024-04-24T01:23:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:23:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:23:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-04-24T01:23:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_10 2024-04-24T01:23:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:23:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:23:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T01:23:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:23:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_10 2024-04-24T01:23:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:23:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T01:23:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-04-24T01:23:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-04-24T01:23:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T01:23:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:23:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-04-24T01:23:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.aEE7yWfHIo/rbd-mirror.test.log 2024-04-24T01:23:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T01:23:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 01:23:04 TEST: check if removed 'images'\''' OMAP are removed 2024-04-24T01:23:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 01:23:04 TEST: check if removed images' OMAP are removed 2024-04-24T01:23:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-04-24T01:23:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:23:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-04-24T01:23:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:23:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirroring 2024-04-24T01:23:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=status_global 2024-04-24T01:23:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T01:23:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=status_global 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c status_global 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T01:23:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-04-24T01:23:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T01:23:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T01:23:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:23:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T01:23:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:23:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-04-24T01:23:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:23:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirroring 2024-04-24T01:23:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=image_ 2024-04-24T01:23:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T01:23:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:23:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:23:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_ 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-04-24T01:23:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_ 2024-04-24T01:23:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T01:23:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T01:23:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:23:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T01:23:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:23:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-04-24T01:23:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:23:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirror_leader 2024-04-24T01:23:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=image_map 2024-04-24T01:23:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T01:23:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:23:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:23:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-04-24T01:23:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:23:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T01:23:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_map 2024-04-24T01:23:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-04-24T01:23:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T01:23:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_map 2024-04-24T01:23:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T01:23:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-04-24T01:23:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T01:23:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T01:23:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirroring 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=status_global 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:23:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:23:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-04-24T01:23:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:23:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T01:23:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=status_global 2024-04-24T01:23:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-04-24T01:23:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:23:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c status_global 2024-04-24T01:23:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T01:23:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirroring 2024-04-24T01:23:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=image_ 2024-04-24T01:23:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T01:23:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_ 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_ 2024-04-24T01:23:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T01:23:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-04-24T01:23:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirror_leader 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=image_map 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T01:23:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T01:23:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_map 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_map 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T01:23:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-04-24T01:23:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T01:23:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T01:23:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T01:23:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T01:23:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T01:23:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cleanup 0 2024-04-24T01:23:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code=0 2024-04-24T01:23:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T01:23:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -ne 0 ']' 2024-04-24T01:23:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z '' ']' 2024-04-24T01:23:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster instance 2024-04-24T01:23:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T01:23:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-24T01:23:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror' does not exist 2024-04-24T01:23:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T01:23:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-24T01:23:07.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.178+0000 7f9ddfa1d640 -1 rbd::mirror::LeaderWatcher: 0x5618168ae600 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-24T01:23:07.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9dd8a0f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x561816031b00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T01:23:07.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9dd8a0f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x561816031680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T01:23:07.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9dd8a0f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5618161cc480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T01:23:07.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9dd9210640 -1 rbd::mirror::PoolWatcher: 0x56181635d860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9dd8a0f640 -1 rbd::mirror::PoolWatcher: 0x56181635d680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9dd8a0f640 -1 rbd::mirror::PoolWatcher: 0x561815fa54a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9dd2202640 -1 rbd::mirror::PoolWatcher: 0x56181635dc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9dd2202640 -1 rbd::mirror::PoolWatcher: 0x561814be32c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9dd2202640 -1 rbd::mirror::PoolWatcher: 0x56181635da40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9ddfa1d640 -1 rbd::mirror::LeaderWatcher: 0x5618168ae600 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-24T01:23:07.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.182+0000 7f9ddfa1d640 -1 rbd::mirror::LeaderWatcher: 0x5618168ae600 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-24T01:23:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror' does not exist 2024-04-24T01:23:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T01:23:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-24T01:23:07.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb15ad57640 -1 rbd::mirror::LeaderWatcher: 0x5578d6752c00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-24T01:23:07.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb15454a640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5578d6c3bb00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T01:23:07.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb15454a640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5578d6954000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T01:23:07.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb15454a640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5578d6d338c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T01:23:07.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb15454a640 -1 rbd::mirror::PoolWatcher: 0x5578d70203c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb15454a640 -1 rbd::mirror::PoolWatcher: 0x5578d7020000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb15454a640 -1 rbd::mirror::PoolWatcher: 0x5578d7020780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb14d53c640 -1 rbd::mirror::PoolWatcher: 0x5578d70212c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb14d53c640 -1 rbd::mirror::PoolWatcher: 0x5578d7020b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.914+0000 7fb14d53c640 -1 rbd::mirror::PoolWatcher: 0x5578d7020f00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:07.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.918+0000 7fb15ad57640 -1 rbd::mirror::LeaderWatcher: 0x5578d6752c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-24T01:23:07.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:07.918+0000 7fb15ad57640 -1 rbd::mirror::LeaderWatcher: 0x5578d6752c00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-24T01:23:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror_parent' does not exist 2024-04-24T01:23:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T01:23:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-24T01:23:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror_parent' does not exist 2024-04-24T01:23:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-24T01:23:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirrors cluster1 2024-04-24T01:23:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T01:23:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:23:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:23:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster1:0 '' 2024-04-24T01:23:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T01:23:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T01:23:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T01:23:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:23:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster1:1 '' 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster1:2 '' 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster1:3 '' 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T01:23:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T01:23:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-24T01:23:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirrors cluster2 2024-04-24T01:23:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T01:23:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster2:0 '' 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:0 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster2:1 '' 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:1 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T01:23:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster2:2 '' 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:2 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster2:3 '' 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:3 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z 1 ']' 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test '' = /tmp/tmp.aEE7yWfHIo 2024-04-24T01:23:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -Rf /tmp/tmp.aEE7yWfHIo 2024-04-24T01:23:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T01:23:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo OK 2024-04-24T01:23:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:OK 2024-04-24T01:23:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ exit 0 2024-04-24T01:23:09.496 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-24T01:23:09.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-04-24T01:23:09.656 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-04-24T01:23:09.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-04-24T01:23:09.922 DEBUG:teuthology.parallel:result is None 2024-04-24T01:23:09.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-24T01:23:09.935 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-24T01:23:09.935 DEBUG:teuthology.orchestra.run.smithi134:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T01:23:09.992 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-24T01:23:09.992 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-24T01:23:10.002 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-24T01:23:10.002 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 20 2024-04-24T01:23:10.002 INFO:tasks.rbd_mirror_thrash:done joining 2024-04-24T01:23:10.002 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-24T01:23:10.039 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-24T01:23:10.039 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 16 2024-04-24T01:23:10.039 INFO:tasks.rbd_mirror_thrash:done joining 2024-04-24T01:23:10.040 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T01:23:10.073 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-04-24T01:23:10.073 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:23:10.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.070+0000 7f9de021e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 100691) UID: 1000 2024-04-24T01:23:10.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.074+0000 7f9ddf21c640 -1 rbd::mirror::InstanceWatcher: 0x561815b5a380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.10855 object: (2) No such file or directory 2024-04-24T01:23:10.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.074+0000 7f9ddf21c640 -1 rbd::mirror::InstanceWatcher: 0x561815b5a700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.10855 object: (2) No such file or directory 2024-04-24T01:23:10.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.074+0000 7f9ddf21c640 -1 rbd::mirror::InstanceWatcher: 0x561816060000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.10857 object: (2) No such file or directory 2024-04-24T01:23:10.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.074+0000 7f9dd8a0f640 -1 rbd::mirror::InstanceWatcher: 0x561815b5a380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:10.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.074+0000 7f9dcb1f4640 -1 rbd::mirror::InstanceWatcher: 0x561816060000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:10.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.074+0000 7f9dd9210640 -1 rbd::mirror::InstanceWatcher: 0x561815b5a700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:10.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9dd2a03640 -1 rbd::mirror::MirrorStatusUpdater 0x561815bd05a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:10.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9dd2a03640 -1 rbd::mirror::MirrorStatusUpdater 0x561815bd0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:10.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9ddfa1d640 -1 rbd::mirror::NamespaceReplayer: 0x5618167aa820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:10.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9ddfa1d640 -1 rbd::mirror::NamespaceReplayer: 0x561815bd2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:10.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9dd8a0f640 -1 rbd::mirror::MirrorStatusUpdater 0x561815bd0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:10.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9dd8a0f640 -1 rbd::mirror::MirrorStatusUpdater 0x561815bd0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:10.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9ddf21c640 -1 rbd::mirror::NamespaceReplayer: 0x5618167aa820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:10.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9ddf21c640 -1 rbd::mirror::NamespaceReplayer: 0x561815bd2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:10.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9ddf21c640 -1 rbd::mirror::LeaderWatcher: 0x5618168ae600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:10.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9ddf21c640 -1 rbd::mirror::InstanceWatcher: 0x561815b5a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.10855 object: (2) No such file or directory 2024-04-24T01:23:10.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.286+0000 7f9dd9210640 -1 rbd::mirror::InstanceWatcher: 0x561815b5a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:10.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.290+0000 7f9dd2202640 -1 rbd::mirror::MirrorStatusUpdater 0x5618167da6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:10.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.290+0000 7f9ddf21c640 -1 rbd::mirror::NamespaceReplayer: 0x561815aec340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:10.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.290+0000 7f9dd8a0f640 -1 rbd::mirror::MirrorStatusUpdater 0x561816738fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:10.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T01:23:10.290+0000 7f9ddfa1d640 -1 rbd::mirror::NamespaceReplayer: 0x561815aec340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:11.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.302+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6321200 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-24T01:23:11.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.302+0000 7fe2d25ba640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5584a6540480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T01:23:11.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.306+0000 7fe2d1db9640 -1 rbd::mirror::PoolWatcher: 0x5584a65ee000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:11.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.306+0000 7fe2cbdad640 -1 rbd::mirror::PoolWatcher: 0x5584a65ef680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:11.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.306+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6321200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-24T01:23:11.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.306+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6321200 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-24T01:23:11.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc803fb5640 -1 rbd::mirror::RemotePoolPoller: 0x55b327e17930 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-24T01:23:11.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc8047b6640 -1 rbd::mirror::RemotePoolPoller: 0x55b327e17930 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-24T01:23:11.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc8047b6640 -1 rbd::mirror::PoolReplayer: 0x55b327e2bb00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-24T01:23:11.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc8117d0640 -1 rbd::mirror::InstanceWatcher: 0x55b328d30380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5435 object: (2) No such file or directory 2024-04-24T01:23:11.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc8117d0640 -1 rbd::mirror::InstanceWatcher: 0x55b328d30700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5435 object: (2) No such file or directory 2024-04-24T01:23:11.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc80afc3640 -1 rbd::mirror::InstanceWatcher: 0x55b328d30380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:11.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc80afc3640 -1 rbd::mirror::InstanceWatcher: 0x55b328d30700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:11.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc803fb5640 -1 rbd::mirror::MirrorStatusUpdater 0x55b328da8360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:11.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc803fb5640 -1 rbd::mirror::MirrorStatusUpdater 0x55b328da85a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:11.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc8117d0640 -1 rbd::mirror::NamespaceReplayer: 0x55b328da6000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:11.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc8117d0640 -1 rbd::mirror::NamespaceReplayer: 0x55b328da61a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:11.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc80afc3640 -1 rbd::mirror::MirrorStatusUpdater 0x55b328da8000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:11.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc80afc3640 -1 rbd::mirror::MirrorStatusUpdater 0x55b328da8120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:11.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc810fcf640 -1 rbd::mirror::NamespaceReplayer: 0x55b328da6000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:11.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.874+0000 7fc810fcf640 -1 rbd::mirror::NamespaceReplayer: 0x55b328da61a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:11.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.946+0000 7fe2d8dc7640 -1 rbd::mirror::RemotePoolPoller: 0x5584a51cb520 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-24T01:23:11.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.946+0000 7fe2d95c8640 -1 rbd::mirror::RemotePoolPoller: 0x5584a51cb520 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-24T01:23:11.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.946+0000 7fe2d95c8640 -1 rbd::mirror::PoolReplayer: 0x5584a51dfb00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-24T01:23:11.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2e65e2640 -1 rbd::mirror::InstanceWatcher: 0x5584a6143c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13922 object: (2) No such file or directory 2024-04-24T01:23:11.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2e65e2640 -1 rbd::mirror::InstanceWatcher: 0x5584a6143880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13922 object: (2) No such file or directory 2024-04-24T01:23:11.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2df5d4640 -1 rbd::mirror::InstanceWatcher: 0x5584a6143c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:11.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2df5d4640 -1 rbd::mirror::InstanceWatcher: 0x5584a6143880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:11.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2d8dc7640 -1 rbd::mirror::MirrorStatusUpdater 0x5584a639ac60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:11.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2d8dc7640 -1 rbd::mirror::MirrorStatusUpdater 0x5584a639aa20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:11.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2e65e2640 -1 rbd::mirror::NamespaceReplayer: 0x5584a65d0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:11.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2e65e2640 -1 rbd::mirror::NamespaceReplayer: 0x5584a65d01a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:11.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2dfdd5640 -1 rbd::mirror::MirrorStatusUpdater 0x5584a639a5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:11.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2dfdd5640 -1 rbd::mirror::MirrorStatusUpdater 0x5584a639a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:11.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2e5de1640 -1 rbd::mirror::NamespaceReplayer: 0x5584a65d0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:11.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:11.950+0000 7fe2e65e2640 -1 rbd::mirror::NamespaceReplayer: 0x5584a65d01a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:12.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.182+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.186+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.190+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.190+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.190+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.190+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.202+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.202+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.202+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.202+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.206+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.206+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.206+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.206+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.210+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.214+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.214+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.214+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.214+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.218+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.218+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.218+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.218+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.222+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.226+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.226+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.226+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.226+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.230+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.230+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.230+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.230+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.238+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.246+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.246+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.246+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.246+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.250+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.250+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.250+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.250+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.258+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.258+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.258+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.258+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.262+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.266+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.266+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.266+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.266+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.274+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.278+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.282+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.282+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.282+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.282+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.286+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.294+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fc8117d0640 -1 rbd::mirror::LeaderWatcher: 0x55b329b3cc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:12.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fc810fcf640 -1 rbd::mirror::InstanceWatcher: 0x55b328d30000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5435 object: (2) No such file or directory 2024-04-24T01:23:12.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fc80a7c2640 -1 rbd::mirror::InstanceWatcher: 0x55b328d30000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:12.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fc8047b6640 -1 rbd::mirror::MirrorStatusUpdater 0x55b32999e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:12.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fc8117d0640 -1 rbd::mirror::NamespaceReplayer: 0x55b328cc8340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:12.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fc80afc3640 -1 rbd::mirror::MirrorStatusUpdater 0x55b329922fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:12.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fc810fcf640 -1 rbd::mirror::NamespaceReplayer: 0x55b328cc8340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:12.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.302+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.306+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.306+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.306+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.306+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fc8117d0640 -1 rbd::mirror::LeaderWatcher: 0x55b328f55200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:12.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fc8117d0640 -1 rbd::mirror::InstanceWatcher: 0x55b328d30a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5439 object: (2) No such file or directory 2024-04-24T01:23:12.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fc7fcfa7640 -1 rbd::mirror::InstanceWatcher: 0x55b328d30a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:12.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fc7f679a640 -1 rbd::mirror::MirrorStatusUpdater 0x55b328f8c7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:12.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fc8117d0640 -1 rbd::mirror::NamespaceReplayer: 0x55b328cc81a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:12.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.310+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fc7fd7a8640 -1 rbd::mirror::MirrorStatusUpdater 0x55b328f810e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:12.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fc810fcf640 -1 rbd::mirror::NamespaceReplayer: 0x55b328cc81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:12.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.314+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.318+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.322+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.338+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fe2cbdad640 -1 rbd::mirror::RemotePoolPoller: 0x5584a51cbe10 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-24T01:23:12.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.342+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fe2cb5ac640 -1 rbd::mirror::RemotePoolPoller: 0x5584a51cbe10 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-24T01:23:12.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fe2cb5ac640 -1 rbd::mirror::PoolReplayer: 0x5584a6f28f00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-24T01:23:12.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.350+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.358+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.370+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.374+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.378+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.386+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.390+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.394+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.398+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.402+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.410+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.414+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.422+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.430+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fe2e5de1640 -1 rbd::mirror::LeaderWatcher: 0x5584a6f28c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:12.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fe2e5de1640 -1 rbd::mirror::InstanceWatcher: 0x5584a6142000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13922 object: (2) No such file or directory 2024-04-24T01:23:12.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fe2df5d4640 -1 rbd::mirror::InstanceWatcher: 0x5584a6142000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:12.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fe2d95c8640 -1 rbd::mirror::MirrorStatusUpdater 0x5584a6d647e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:12.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fe2e5de1640 -1 rbd::mirror::NamespaceReplayer: 0x5584a60a4340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:12.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fe2dfdd5640 -1 rbd::mirror::MirrorStatusUpdater 0x5584a6cf4fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:12.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fe2e65e2640 -1 rbd::mirror::NamespaceReplayer: 0x5584a60a4340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:12.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fe2e65e2640 -1 rbd::mirror::LeaderWatcher: 0x5584a6321200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:12.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fe2e65e2640 -1 rbd::mirror::InstanceWatcher: 0x5584a65f8000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13924 object: (2) No such file or directory 2024-04-24T01:23:12.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fe2d1db9640 -1 rbd::mirror::InstanceWatcher: 0x5584a65f8000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:12.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fe2cbdad640 -1 rbd::mirror::MirrorStatusUpdater 0x5584a639ab40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:12.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fe2e5de1640 -1 rbd::mirror::NamespaceReplayer: 0x5584a60a4680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:12.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fe2d25ba640 -1 rbd::mirror::MirrorStatusUpdater 0x5584a637e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:12.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:12.442+0000 7fe2e5de1640 -1 rbd::mirror::NamespaceReplayer: 0x5584a60a4680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:12.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e02394000 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-24T01:23:12.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f0a85c5c640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x557e02d20d80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T01:23:12.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f0a8645d640 -1 rbd::mirror::PoolWatcher: 0x557e02759860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:12.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f0a7f44f640 -1 rbd::mirror::PoolWatcher: 0x557e012ad2c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T01:23:12.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e02394000 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-24T01:23:12.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e02394000 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-24T01:23:12.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:12.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:12.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:12.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.166+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.210+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.270+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:13.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:13.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.522+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.638+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:14.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:14.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:13.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:13.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.090+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.250+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.266+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.306+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.318+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:15.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:15.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.362+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-04-24T01:23:16.175 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T01:23:16.189 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-04-24T01:23:16.189 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:23:16.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.550+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.682+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:16.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:16.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.826+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.854+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.942+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.950+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.962+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:14.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:14.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.038+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.066+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.166+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.202+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.206+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.210+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.210+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.214+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.218+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.222+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.226+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.230+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.234+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.242+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.246+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.250+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.254+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.262+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.266+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.270+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.274+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.282+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.286+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.290+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.294+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:17.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:17.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.306+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.310+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.314+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.318+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.322+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.326+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.338+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.342+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.346+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.354+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.358+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.362+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.366+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.370+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.374+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.378+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.386+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.394+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.398+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.402+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.418+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.422+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.426+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.430+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.434+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.438+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.442+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.446+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.450+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.458+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.462+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.466+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.470+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.474+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.478+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.482+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.498+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.506+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.510+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.514+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.530+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.534+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.546+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.550+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.554+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.558+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.562+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.566+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.570+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.574+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.578+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.582+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.590+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.594+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.598+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.606+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.610+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.618+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.622+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.630+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.634+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.642+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.646+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.650+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.654+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.658+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.662+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.666+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.670+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.674+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.678+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.682+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.686+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.690+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.694+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.698+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.698+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.706+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.714+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.722+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.730+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.738+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.746+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.754+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.758+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.762+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.766+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.770+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.774+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.778+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.782+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.786+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.790+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.794+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.802+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.806+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.810+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.814+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.818+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:18.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:18.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.826+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.830+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.834+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.838+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.842+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.846+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.854+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.858+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.862+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.866+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.870+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.874+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.878+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.882+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.886+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.890+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.894+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.898+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.902+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.906+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.910+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.914+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.918+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.922+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.922+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.922+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.926+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.930+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.934+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.938+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.942+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.946+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.950+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb15a556640 -1 rbd::mirror::InstanceWatcher: 0x5578d6804380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4939 object: (2) No such file or directory 2024-04-24T01:23:19.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb15a556640 -1 rbd::mirror::InstanceWatcher: 0x5578d6804a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4939 object: (2) No such file or directory 2024-04-24T01:23:19.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb15454a640 -1 rbd::mirror::InstanceWatcher: 0x5578d6804380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:19.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb153d49640 -1 rbd::mirror::InstanceWatcher: 0x5578d6804a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:19.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb14dd3d640 -1 rbd::mirror::MirrorStatusUpdater 0x5578d68745a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb14dd3d640 -1 rbd::mirror::MirrorStatusUpdater 0x5578d6874360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb15a556640 -1 rbd::mirror::NamespaceReplayer: 0x5578d6872000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb15a556640 -1 rbd::mirror::NamespaceReplayer: 0x5578d68721a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb15454a640 -1 rbd::mirror::MirrorStatusUpdater 0x5578d6874000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb15a556640 -1 rbd::mirror::NamespaceReplayer: 0x5578d6872000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb153d49640 -1 rbd::mirror::MirrorStatusUpdater 0x5578d6874120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fb15a556640 -1 rbd::mirror::NamespaceReplayer: 0x5578d68721a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.954+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.958+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.962+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.962+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.966+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.970+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.974+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.978+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.982+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.986+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.990+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.994+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:15.998+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:15.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.002+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.006+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.010+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.014+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.018+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.022+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.026+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.030+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.034+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.038+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.042+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.046+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.050+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.054+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.058+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.062+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.066+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.070+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.074+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.078+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.082+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.086+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.090+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.094+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.098+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.106+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.110+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.110+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.114+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.118+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.122+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.126+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.130+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.134+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.138+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.142+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.150+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.154+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.162+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.166+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.170+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.174+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.178+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.182+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.186+0000 7fd444a13640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 126769) UID: 1000 2024-04-24T01:23:19.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625d33200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:19.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::InstanceWatcher: 0x561625b0e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.12947 object: (2) No such file or directory 2024-04-24T01:23:19.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd443a11640 -1 rbd::mirror::InstanceWatcher: 0x561625b0e700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.12947 object: (2) No such file or directory 2024-04-24T01:23:19.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd43d204640 -1 rbd::mirror::InstanceWatcher: 0x561625b0e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:19.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd42f9e9640 -1 rbd::mirror::InstanceWatcher: 0x561625fe8000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:19.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd43da05640 -1 rbd::mirror::InstanceWatcher: 0x561625b0e700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:19.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd4301ea640 -1 rbd::mirror::MirrorStatusUpdater 0x561625da9320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::NamespaceReplayer: 0x561625b6c4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd4371f8640 -1 rbd::mirror::MirrorStatusUpdater 0x561625b6e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::NamespaceReplayer: 0x561625b6c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd4369f7640 -1 rbd::mirror::MirrorStatusUpdater 0x561625b6e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::NamespaceReplayer: 0x561625b6c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd43d204640 -1 rbd::mirror::MirrorStatusUpdater 0x561625b6e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::NamespaceReplayer: 0x561625b6c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd43da05640 -1 rbd::mirror::MirrorStatusUpdater 0x561625b6e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::NamespaceReplayer: 0x561625b6c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.190+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7fd444212640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7fd443a11640 -1 rbd::mirror::LeaderWatcher: 0x561625a6cc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:19.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7fd443a11640 -1 rbd::mirror::InstanceWatcher: 0x561625b0e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.12947 object: (2) No such file or directory 2024-04-24T01:23:19.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7fd43d204640 -1 rbd::mirror::InstanceWatcher: 0x561625b0e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:19.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7fd4371f8640 -1 rbd::mirror::MirrorStatusUpdater 0x5616265227e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7fd443a11640 -1 rbd::mirror::NamespaceReplayer: 0x5616258861a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7fd43da05640 -1 rbd::mirror::MirrorStatusUpdater 0x5616265210e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7fd444212640 -1 rbd::mirror::NamespaceReplayer: 0x5616258861a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.214+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.234+0000 7fb13fd21640 -1 rbd::mirror::RemotePoolPoller: 0x5578d56c7ee0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-24T01:23:19.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.234+0000 7fb140522640 -1 rbd::mirror::RemotePoolPoller: 0x5578d56c7ee0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-24T01:23:19.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.234+0000 7fb140522640 -1 rbd::mirror::PoolReplayer: 0x5578d6752f00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-24T01:23:19.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.254+0000 7fb14dd3d640 -1 rbd::mirror::RemotePoolPoller: 0x5578d56c75f0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-24T01:23:19.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.254+0000 7fb14d53c640 -1 rbd::mirror::RemotePoolPoller: 0x5578d56c75f0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-24T01:23:19.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.254+0000 7fb14d53c640 -1 rbd::mirror::PoolReplayer: 0x5578d56dbb00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-24T01:23:19.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.262+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.298+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.322+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.338+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a97c80640 -1 rbd::mirror::InstanceWatcher: 0x557e02cc5500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4312 object: (2) No such file or directory 2024-04-24T01:23:19.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a9747f640 -1 rbd::mirror::InstanceWatcher: 0x557e02cc5c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4312 object: (2) No such file or directory 2024-04-24T01:23:19.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a93477640 -1 rbd::mirror::InstanceWatcher: 0x557e02cc5500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:19.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a93477640 -1 rbd::mirror::InstanceWatcher: 0x557e02cc5c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:19.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a8ac66640 -1 rbd::mirror::MirrorStatusUpdater 0x557e02e81320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a8ac66640 -1 rbd::mirror::MirrorStatusUpdater 0x557e02e80ea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a9747f640 -1 rbd::mirror::NamespaceReplayer: 0x557e0275e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a9747f640 -1 rbd::mirror::NamespaceReplayer: 0x557e0287ed00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a93477640 -1 rbd::mirror::MirrorStatusUpdater 0x557e02e80b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a93477640 -1 rbd::mirror::MirrorStatusUpdater 0x557e02e80c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:19.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a97c80640 -1 rbd::mirror::NamespaceReplayer: 0x557e0275e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a97c80640 -1 rbd::mirror::NamespaceReplayer: 0x557e0287ed00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:19.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:19.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:19.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.406+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.446+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.478+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.482+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.486+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.502+0000 7fb15a556640 -1 rbd::mirror::LeaderWatcher: 0x5578d6752c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:20.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.502+0000 7fb15a556640 -1 rbd::mirror::InstanceWatcher: 0x5578d6804000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4939 object: (2) No such file or directory 2024-04-24T01:23:20.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.502+0000 7fb15454a640 -1 rbd::mirror::InstanceWatcher: 0x5578d6804000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:20.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.502+0000 7fb14dd3d640 -1 rbd::mirror::MirrorStatusUpdater 0x5578d72546c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:20.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.502+0000 7fb15a556640 -1 rbd::mirror::NamespaceReplayer: 0x5578d65781a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:20.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.502+0000 7fb153d49640 -1 rbd::mirror::MirrorStatusUpdater 0x5578d71e4fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:20.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.502+0000 7fb15ad57640 -1 rbd::mirror::NamespaceReplayer: 0x5578d65781a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:20.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.506+0000 7fb15ad57640 -1 rbd::mirror::LeaderWatcher: 0x5578d6a11200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:20.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.510+0000 7fb15ad57640 -1 rbd::mirror::InstanceWatcher: 0x5578d6804700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4942 object: (2) No such file or directory 2024-04-24T01:23:20.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.510+0000 7fb14652e640 -1 rbd::mirror::InstanceWatcher: 0x5578d6804700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:20.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.510+0000 7fb13fd21640 -1 rbd::mirror::MirrorStatusUpdater 0x5578d6a7e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:20.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.510+0000 7fb15ad57640 -1 rbd::mirror::NamespaceReplayer: 0x5578d6cea000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:20.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.510+0000 7fb146d2f640 -1 rbd::mirror::MirrorStatusUpdater 0x5578d6a4b200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:20.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:16.510+0000 7fb15a556640 -1 rbd::mirror::NamespaceReplayer: 0x5578d6cea000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:20.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.538+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.554+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.562+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.570+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.614+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a7f44f640 -1 rbd::mirror::RemotePoolPoller: 0x557e012e7e10 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-24T01:23:20.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a7dc4c640 -1 rbd::mirror::RemotePoolPoller: 0x557e012e7e10 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-24T01:23:20.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a7dc4c640 -1 rbd::mirror::PoolReplayer: 0x557e02394600 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-24T01:23:20.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.622+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.626+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.634+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.638+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.646+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.658+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.662+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.666+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.670+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.678+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.718+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.722+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.738+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.742+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.758+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.770+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.782+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.790+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.794+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.798+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.802+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.806+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.826+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.830+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.834+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.838+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.850+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.878+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.898+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.902+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.914+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.938+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.942+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.946+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.950+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.954+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.962+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.978+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.986+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.994+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:16.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.006+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:20.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:20.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.014+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.018+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.022+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.030+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.034+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.050+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.054+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.062+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.066+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.070+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.106+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.134+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.138+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.142+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.146+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.150+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.162+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.166+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.174+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.186+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.190+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.194+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.202+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.206+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.226+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.234+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.238+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.246+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.254+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.258+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.262+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.282+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.290+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.294+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.302+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.306+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.314+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.318+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.326+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.330+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.338+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.342+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.346+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.358+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.362+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.378+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.382+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.386+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.390+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.394+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.402+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.406+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.414+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.418+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.426+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.430+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f0a97c80640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e0266cf00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:21.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f0a97c80640 -1 rbd::mirror::InstanceWatcher: 0x557e026f0a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4312 object: (2) No such file or directory 2024-04-24T01:23:21.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f0a93477640 -1 rbd::mirror::InstanceWatcher: 0x557e026f0a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:21.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f0a8c469640 -1 rbd::mirror::MirrorStatusUpdater 0x557e025e0ea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:21.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f0a97c80640 -1 rbd::mirror::NamespaceReplayer: 0x557e023fdd40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:21.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f0a92c76640 -1 rbd::mirror::MirrorStatusUpdater 0x557e021e45a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:21.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f0a9747f640 -1 rbd::mirror::NamespaceReplayer: 0x557e023fdd40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:21.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f0a9747f640 -1 rbd::mirror::LeaderWatcher: 0x557e02394000 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:21.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f0a9747f640 -1 rbd::mirror::InstanceWatcher: 0x557e02cc5180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4314 object: (2) No such file or directory 2024-04-24T01:23:21.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f0a8645d640 -1 rbd::mirror::InstanceWatcher: 0x557e02cc5180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:21.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f0a7f44f640 -1 rbd::mirror::MirrorStatusUpdater 0x557e02625d40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:21.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f0a97c80640 -1 rbd::mirror::NamespaceReplayer: 0x557e0287e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:21.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f0a85c5c640 -1 rbd::mirror::MirrorStatusUpdater 0x557e02624480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:21.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f0a9747f640 -1 rbd::mirror::NamespaceReplayer: 0x557e0287e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:21.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.490+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.494+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.526+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.538+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.542+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.546+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.550+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.554+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.558+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.562+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.566+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.570+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.574+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.578+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.582+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.586+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.590+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.594+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.598+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.602+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.606+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.610+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.614+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.618+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.622+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.626+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.630+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.634+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.638+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.642+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.646+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.650+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.654+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.658+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.662+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.666+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.670+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.674+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.678+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.682+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.686+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.690+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.694+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.698+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.702+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.706+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.710+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.714+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.718+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.722+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.726+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.730+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.734+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.738+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.742+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.746+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.750+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.754+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.758+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.762+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.766+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.770+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.774+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.778+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.782+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.786+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.790+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.794+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.798+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.802+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.806+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.810+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.814+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.818+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.822+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.826+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.830+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.834+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.838+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.842+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.846+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.850+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.854+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.858+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.862+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.866+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.870+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.874+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.878+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.882+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.886+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.890+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.894+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.898+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.902+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.906+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.910+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.914+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.918+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.922+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.926+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.930+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.934+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.938+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.942+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.946+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.950+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.954+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.958+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.962+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.966+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.970+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.974+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.978+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.982+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.986+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.990+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.994+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:17.998+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.002+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.006+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.010+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.014+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.018+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.022+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.026+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.030+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.034+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.038+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.042+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.046+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.050+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.054+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.058+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.062+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.066+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.070+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.074+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.078+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.082+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.086+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.090+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.094+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.098+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.102+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.106+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:21.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:21.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.110+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.114+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.118+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.122+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.126+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.130+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.134+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.138+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.142+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.146+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.150+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.154+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.158+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.162+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.166+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.170+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.174+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.178+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.182+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.186+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.190+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.194+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.198+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.202+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.206+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.210+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.214+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.218+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.222+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.226+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.230+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.234+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.238+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.242+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.246+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.250+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.254+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.258+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.262+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.266+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.270+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.274+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.278+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.282+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.286+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.290+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.294+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.298+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.302+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.306+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.310+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.314+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.318+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.322+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.326+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.330+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.334+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.338+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.342+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.346+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.350+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.354+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.358+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.362+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.366+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.370+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.374+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.378+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.382+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.386+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.390+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.394+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.398+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.402+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.406+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.410+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.414+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.418+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.422+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.426+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.430+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-04-24T01:23:22.295 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T01:23:22.408 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-04-24T01:23:22.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:23:22.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.434+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.438+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.442+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.446+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.450+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.454+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.458+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.462+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.466+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.470+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.474+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.478+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.482+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.486+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.490+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.494+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.498+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.502+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.506+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.510+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.514+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.518+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.522+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.526+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.530+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.534+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.538+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.549+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.549+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.673+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.705+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.741+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:18.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.033+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.057+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:22.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:22.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.229+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.437+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.521+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.573+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.657+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.669+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.673+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.697+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.701+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.901+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:19.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:23.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:23.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.097+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.433+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.445+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.461+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.465+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.473+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.505+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.509+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.585+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.605+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.617+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.621+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.629+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.641+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.661+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.665+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.689+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.693+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.925+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.941+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.949+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.953+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:24.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.969+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.973+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.009+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.017+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.029+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.037+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.037+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.037+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.061+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.069+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.069+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.069+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.069+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.069+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.069+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.073+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.077+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.089+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.089+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.089+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.089+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.089+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.089+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.105+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.109+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.133+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.133+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.133+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.133+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.133+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.133+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.133+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.133+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:20.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.189+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.221+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.221+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.333+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.385+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.613+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.617+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.621+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.621+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.621+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.621+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.621+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.621+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.621+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.621+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.625+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.629+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.633+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.637+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.641+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.645+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.649+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.653+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.657+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.661+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.665+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.669+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.685+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.693+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.697+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.701+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.761+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.765+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.785+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.793+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.797+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.801+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.805+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.809+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.813+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.817+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.825+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.829+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.837+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.841+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.845+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.889+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.893+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.009+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:21.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.025+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.029+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.041+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.053+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.077+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.085+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.093+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.105+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.109+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.121+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.129+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.137+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.145+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.161+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.165+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.169+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.173+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.177+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.181+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.185+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.197+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.209+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.217+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.237+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.245+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.783 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.261+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.269+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.273+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.281+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.289+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.293+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.317+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.321+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.329+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.333+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.337+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.341+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f6f5fd7b640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f6f6057c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 140605) UID: 1000 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5fd7b640 -1 rbd::mirror::InstanceWatcher: 0x55ffaa758380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13979 object: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5f57a640 -1 rbd::mirror::InstanceWatcher: 0x55ffaa758a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13979 object: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f4bd53640 -1 rbd::mirror::InstanceWatcher: 0x55ffaa758700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f58d6d640 -1 rbd::mirror::InstanceWatcher: 0x55ffaa758380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f58d6d640 -1 rbd::mirror::InstanceWatcher: 0x55ffaa758a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f52560640 -1 rbd::mirror::MirrorStatusUpdater 0x55ffaa7da5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:25.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5f57a640 -1 rbd::mirror::NamespaceReplayer: 0x55ffaa7d8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f52d61640 -1 rbd::mirror::MirrorStatusUpdater 0x55ffaa7da360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5f57a640 -1 rbd::mirror::NamespaceReplayer: 0x55ffaa7d81a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f58d6d640 -1 rbd::mirror::MirrorStatusUpdater 0x55ffaa7da000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f58d6d640 -1 rbd::mirror::MirrorStatusUpdater 0x55ffaa7da120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5fd7b640 -1 rbd::mirror::NamespaceReplayer: 0x55ffaa7d8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5fd7b640 -1 rbd::mirror::NamespaceReplayer: 0x55ffaa7d81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f6f5f57a640 -1 rbd::mirror::LeaderWatcher: 0x55ffaa6ccc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f6f5f57a640 -1 rbd::mirror::InstanceWatcher: 0x55ffaa758000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.13979 object: (2) No such file or directory 2024-04-24T01:23:25.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f6f58d6d640 -1 rbd::mirror::InstanceWatcher: 0x55ffaa758000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:25.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f6f52560640 -1 rbd::mirror::MirrorStatusUpdater 0x55ffaa51c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:25.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f6f5f57a640 -1 rbd::mirror::NamespaceReplayer: 0x55ffab062000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:25.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f6f5956e640 -1 rbd::mirror::MirrorStatusUpdater 0x55ffab15d0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:25.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f6f5fd7b640 -1 rbd::mirror::NamespaceReplayer: 0x55ffab062000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:22.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:23.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:24.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:25.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.173+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.225+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.649+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.709+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.757+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.825+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.929+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.961+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:26.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.293+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.297+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.313+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.349+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.445+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.485+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.553+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.597+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.605+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.641+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.645+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.657+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.661+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.665+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.669+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.673+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.677+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.681+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.685+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.689+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.693+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.697+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.701+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.705+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.709+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.713+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.717+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.717+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.721+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.725+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.729+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.733+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.737+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.737+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.741+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.745+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.749+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.753+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.757+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.761+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.761+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.765+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.769+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.773+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.777+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.781+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.785+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.789+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.793+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.793+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.797+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.801+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.805+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.809+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.813+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.817+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.821+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.821+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.825+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.829+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.833+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.837+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.841+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.845+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.849+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.849+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.853+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.857+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.857+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.861+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.861+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.865+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.869+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.873+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.877+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.881+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.885+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.889+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.893+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.897+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.901+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.905+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.909+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.913+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.917+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.921+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.921+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.925+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.929+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.933+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.937+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.941+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.945+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.949+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.949+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.953+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.957+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.957+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.961+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.965+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.969+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.973+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.977+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.981+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.985+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.985+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.989+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.993+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:27.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:27.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:27.997+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.001+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.005+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.005+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.009+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.013+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.017+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.021+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.021+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.025+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.025+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.029+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.033+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.037+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.041+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.045+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.049+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.049+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.053+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.053+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.057+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.061+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.061+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.065+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.069+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.069+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.073+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.077+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.081+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.085+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.089+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.093+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.097+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.101+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.105+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.109+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.113+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.117+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.121+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.125+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.129+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.129+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.133+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.137+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.141+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.145+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.149+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.153+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.157+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.161+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.165+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.169+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.173+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.177+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.181+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.185+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.189+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.193+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.197+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.201+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.205+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.209+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.209+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.213+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.217+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.221+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.221+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.225+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.229+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.233+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.237+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.241+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.245+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.249+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.249+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.253+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.253+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.257+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.257+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.261+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.261+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.265+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.265+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.269+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.273+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.277+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.281+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.285+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.285+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.289+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.293+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.297+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.301+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.305+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.309+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.313+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.317+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.321+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.321+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.325+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.329+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.333+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.337+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.337+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.341+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.345+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.349+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.353+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.357+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.361+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.365+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.369+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.373+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.377+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.381+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.385+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.385+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.389+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.393+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.397+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.401+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.405+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.409+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.413+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.417+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.421+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.425+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.429+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.433+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.437+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.441+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.441+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.445+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.449+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.453+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.457+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.461+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.461+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.465+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.469+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.469+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.473+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.477+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.481+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.485+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.489+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.493+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.497+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.497+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.501+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.505+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.505+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-04-24T01:23:28.511 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T01:23:28.522 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-04-24T01:23:28.522 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.509+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.513+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.517+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.517+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T01:23:28.521+0000 7fe2e6de3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 139796) UID: 1000 2024-04-24T01:23:28.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.521+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.525+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.529+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.533+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.537+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.541+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-04-24T01:23:28.550 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T01:23:28.561 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-04-24T01:23:28.562 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:23:28.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.545+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.549+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.553+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.557+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T01:23:28.561+0000 7fb15b558640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 119803) UID: 1000 2024-04-24T01:23:28.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.561+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.561+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.565+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.569+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.573+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.577+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.581+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.585+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-04-24T01:23:28.593 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T01:23:28.603 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-04-24T01:23:28.603 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:23:28.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.589+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.593+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.597+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f0a98481640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 30181) UID: 1000 2024-04-24T01:23:28.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.601+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.605+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.609+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.613+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.613+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.617+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.621+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.625+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-04-24T01:23:28.634 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T01:23:28.644 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-04-24T01:23:28.644 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:23:28.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.629+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.633+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.637+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.641+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.641+0000 7f8036d2d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 130139) UID: 1000 2024-04-24T01:23:28.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::InstanceWatcher: 0x55f5ffa7c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5055 object: (2) No such file or directory 2024-04-24T01:23:28.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::InstanceWatcher: 0x55f5ffa7ca80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5055 object: (2) No such file or directory 2024-04-24T01:23:28.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ffc9f200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:28.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f802f51e640 -1 rbd::mirror::InstanceWatcher: 0x55f5ffa7c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:28.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f802f51e640 -1 rbd::mirror::InstanceWatcher: 0x55f5ffa7ca80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:28.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8028d11640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5ffaea5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:28.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8028d11640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5ffaea360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:28.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f803652c640 -1 rbd::mirror::NamespaceReplayer: 0x55f5ffae8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f803652c640 -1 rbd::mirror::NamespaceReplayer: 0x55f5ffae81a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::InstanceWatcher: 0x55f5fff4e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5058 object: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f802f51e640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5ffaea000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f803652c640 -1 rbd::mirror::NamespaceReplayer: 0x55f5ffae8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f802f51e640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5ffaea120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f803652c640 -1 rbd::mirror::NamespaceReplayer: 0x55f5ffae81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8021d03640 -1 rbd::mirror::InstanceWatcher: 0x55f5fff4e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8022504640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5ffd17440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.645+0000 7f8035d2b640 -1 rbd::mirror::NamespaceReplayer: 0x55f5ffae84e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.649+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-24T01:23:28.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.653+0000 7f8035d2b640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-24T01:23:28.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.653+0000 7f803652c640 -1 rbd::mirror::LeaderWatcher: 0x55f5ff9e6c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T01:23:28.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.653+0000 7f803652c640 -1 rbd::mirror::InstanceWatcher: 0x55f5ffa7c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5055 object: (2) No such file or directory 2024-04-24T01:23:28.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.653+0000 7f802f51e640 -1 rbd::mirror::InstanceWatcher: 0x55f5ffa7c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T01:23:28.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.653+0000 7f8029512640 -1 rbd::mirror::MirrorStatusUpdater 0x55f6004d06c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:28.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.653+0000 7f803652c640 -1 rbd::mirror::NamespaceReplayer: 0x55f5ff7ee1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T01:23:28.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.653+0000 7f802fd1f640 -1 rbd::mirror::MirrorStatusUpdater 0x55f6004610e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T01:23:28.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T01:23:28.653+0000 7f8035d2b640 -1 rbd::mirror::NamespaceReplayer: 0x55f5ff7ee1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T01:23:28.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-04-24T01:23:28.686 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T01:23:28.697 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-04-24T01:23:28.697 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:23:28.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T01:23:28.729+0000 7fc811fd1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 139497) UID: 1000 2024-04-24T01:23:28.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-04-24T01:23:28.754 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T01:23:28.765 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-24T01:23:28.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T01:23:29.103 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T01:23:29.104 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T01:23:29.122 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1773,"stamp":"2024-04-24T01:23:29.100246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4639424,"kb_used_data":15128,"kb_used_omap":463,"kb_used_meta":4623792,"kb_avail":278476096,"statfs":{"total":289910292480,"available":285159522304,"internally_reserved":0,"allocated":15491072,"data_stored":7936265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":474135,"internal_metadata":4734764009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003255"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366853+0000","last_change":"2024-04-24T00:37:47.429755+0000","last_active":"2024-04-24T01:23:09.366853+0000","last_peered":"2024-04-24T01:23:09.366853+0000","last_clean":"2024-04-24T01:23:09.366853+0000","last_became_active":"2024-04-24T00:37:47.429213+0000","last_became_peered":"2024-04-24T00:37:47.429213+0000","last_unstale":"2024-04-24T01:23:09.366853+0000","last_undegraded":"2024-04-24T01:23:09.366853+0000","last_fullsized":"2024-04-24T01:23:09.366853+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:34:19.765148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366899+0000","last_change":"2024-04-24T00:37:47.427816+0000","last_active":"2024-04-24T01:23:09.366899+0000","last_peered":"2024-04-24T01:23:09.366899+0000","last_clean":"2024-04-24T01:23:09.366899+0000","last_became_active":"2024-04-24T00:37:47.427410+0000","last_became_peered":"2024-04-24T00:37:47.427410+0000","last_unstale":"2024-04-24T01:23:09.366899+0000","last_undegraded":"2024-04-24T01:23:09.366899+0000","last_fullsized":"2024-04-24T01:23:09.366899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:05:01.512170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366640+0000","last_change":"2024-04-24T00:37:47.428837+0000","last_active":"2024-04-24T01:23:09.366640+0000","last_peered":"2024-04-24T01:23:09.366640+0000","last_clean":"2024-04-24T01:23:09.366640+0000","last_became_active":"2024-04-24T00:37:47.428266+0000","last_became_peered":"2024-04-24T00:37:47.428266+0000","last_unstale":"2024-04-24T01:23:09.366640+0000","last_undegraded":"2024-04-24T01:23:09.366640+0000","last_fullsized":"2024-04-24T01:23:09.366640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:43.968711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.361620+0000","last_change":"2024-04-24T00:37:47.429020+0000","last_active":"2024-04-24T01:23:09.361620+0000","last_peered":"2024-04-24T01:23:09.361620+0000","last_clean":"2024-04-24T01:23:09.361620+0000","last_became_active":"2024-04-24T00:37:47.427471+0000","last_became_peered":"2024-04-24T00:37:47.427471+0000","last_unstale":"2024-04-24T01:23:09.361620+0000","last_undegraded":"2024-04-24T01:23:09.361620+0000","last_fullsized":"2024-04-24T01:23:09.361620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:41.046954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":805,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.503562+0000","last_change":"2024-04-24T00:37:47.427475+0000","last_active":"2024-04-24T01:23:09.503562+0000","last_peered":"2024-04-24T01:23:09.503562+0000","last_clean":"2024-04-24T01:23:09.503562+0000","last_became_active":"2024-04-24T00:37:47.426947+0000","last_became_peered":"2024-04-24T00:37:47.426947+0000","last_unstale":"2024-04-24T01:23:09.503562+0000","last_undegraded":"2024-04-24T01:23:09.503562+0000","last_fullsized":"2024-04-24T01:23:09.503562+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:09:53.202558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.138388+0000","last_change":"2024-04-24T00:37:47.431766+0000","last_active":"2024-04-24T01:23:09.138388+0000","last_peered":"2024-04-24T01:23:09.138388+0000","last_clean":"2024-04-24T01:23:09.138388+0000","last_became_active":"2024-04-24T00:37:47.431433+0000","last_became_peered":"2024-04-24T00:37:47.431433+0000","last_unstale":"2024-04-24T01:23:09.138388+0000","last_undegraded":"2024-04-24T01:23:09.138388+0000","last_fullsized":"2024-04-24T01:23:09.138388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:29:01.749521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":805,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.080489+0000","last_change":"2024-04-24T00:37:47.422610+0000","last_active":"2024-04-24T01:23:09.080489+0000","last_peered":"2024-04-24T01:23:09.080489+0000","last_clean":"2024-04-24T01:23:09.080489+0000","last_became_active":"2024-04-24T00:37:47.422197+0000","last_became_peered":"2024-04-24T00:37:47.422197+0000","last_unstale":"2024-04-24T01:23:09.080489+0000","last_undegraded":"2024-04-24T01:23:09.080489+0000","last_fullsized":"2024-04-24T01:23:09.080489+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:02:16.063327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.367103+0000","last_change":"2024-04-24T00:37:47.430799+0000","last_active":"2024-04-24T01:23:09.367103+0000","last_peered":"2024-04-24T01:23:09.367103+0000","last_clean":"2024-04-24T01:23:09.367103+0000","last_became_active":"2024-04-24T00:37:47.430316+0000","last_became_peered":"2024-04-24T00:37:47.430316+0000","last_unstale":"2024-04-24T01:23:09.367103+0000","last_undegraded":"2024-04-24T01:23:09.367103+0000","last_fullsized":"2024-04-24T01:23:09.367103+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:25:37.002437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":937,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.358159+0000","last_change":"2024-04-24T00:37:43.400334+0000","last_active":"2024-04-24T01:23:09.358159+0000","last_peered":"2024-04-24T01:23:09.358159+0000","last_clean":"2024-04-24T01:23:09.358159+0000","last_became_active":"2024-04-24T00:37:43.399665+0000","last_became_peered":"2024-04-24T00:37:43.399665+0000","last_unstale":"2024-04-24T01:23:09.358159+0000","last_undegraded":"2024-04-24T01:23:09.358159+0000","last_fullsized":"2024-04-24T01:23:09.358159+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_clean_scrub_stamp":"2024-04-24T00:37:42.383719+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:35:49.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738923,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1459644,"kb_used_data":4656,"kb_used_omap":142,"kb_used_meta":1454833,"kb_avail":92912196,"statfs":{"total":96636764160,"available":95142088704,"internally_reserved":0,"allocated":4767744,"data_stored":2251843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":146114,"internal_metadata":1489749310},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738921,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1614784,"kb_used_data":5236,"kb_used_omap":124,"kb_used_meta":1609411,"kb_avail":92757056,"statfs":{"total":96636764160,"available":94983225344,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127727,"internal_metadata":1648037137},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738923,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1564996,"kb_used_data":5236,"kb_used_omap":195,"kb_used_meta":1559548,"kb_avail":92806844,"statfs":{"total":96636764160,"available":95034208256,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200294,"internal_metadata":1596977562},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:23:29.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T01:23:29.425 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T01:23:29.425 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T01:23:29.444 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1773,"stamp":"2024-04-24T01:23:29.100246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4639424,"kb_used_data":15128,"kb_used_omap":463,"kb_used_meta":4623792,"kb_avail":278476096,"statfs":{"total":289910292480,"available":285159522304,"internally_reserved":0,"allocated":15491072,"data_stored":7936265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":474135,"internal_metadata":4734764009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003255"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366853+0000","last_change":"2024-04-24T00:37:47.429755+0000","last_active":"2024-04-24T01:23:09.366853+0000","last_peered":"2024-04-24T01:23:09.366853+0000","last_clean":"2024-04-24T01:23:09.366853+0000","last_became_active":"2024-04-24T00:37:47.429213+0000","last_became_peered":"2024-04-24T00:37:47.429213+0000","last_unstale":"2024-04-24T01:23:09.366853+0000","last_undegraded":"2024-04-24T01:23:09.366853+0000","last_fullsized":"2024-04-24T01:23:09.366853+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:34:19.765148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366899+0000","last_change":"2024-04-24T00:37:47.427816+0000","last_active":"2024-04-24T01:23:09.366899+0000","last_peered":"2024-04-24T01:23:09.366899+0000","last_clean":"2024-04-24T01:23:09.366899+0000","last_became_active":"2024-04-24T00:37:47.427410+0000","last_became_peered":"2024-04-24T00:37:47.427410+0000","last_unstale":"2024-04-24T01:23:09.366899+0000","last_undegraded":"2024-04-24T01:23:09.366899+0000","last_fullsized":"2024-04-24T01:23:09.366899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:05:01.512170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366640+0000","last_change":"2024-04-24T00:37:47.428837+0000","last_active":"2024-04-24T01:23:09.366640+0000","last_peered":"2024-04-24T01:23:09.366640+0000","last_clean":"2024-04-24T01:23:09.366640+0000","last_became_active":"2024-04-24T00:37:47.428266+0000","last_became_peered":"2024-04-24T00:37:47.428266+0000","last_unstale":"2024-04-24T01:23:09.366640+0000","last_undegraded":"2024-04-24T01:23:09.366640+0000","last_fullsized":"2024-04-24T01:23:09.366640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:43.968711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.361620+0000","last_change":"2024-04-24T00:37:47.429020+0000","last_active":"2024-04-24T01:23:09.361620+0000","last_peered":"2024-04-24T01:23:09.361620+0000","last_clean":"2024-04-24T01:23:09.361620+0000","last_became_active":"2024-04-24T00:37:47.427471+0000","last_became_peered":"2024-04-24T00:37:47.427471+0000","last_unstale":"2024-04-24T01:23:09.361620+0000","last_undegraded":"2024-04-24T01:23:09.361620+0000","last_fullsized":"2024-04-24T01:23:09.361620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:41.046954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":805,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.503562+0000","last_change":"2024-04-24T00:37:47.427475+0000","last_active":"2024-04-24T01:23:09.503562+0000","last_peered":"2024-04-24T01:23:09.503562+0000","last_clean":"2024-04-24T01:23:09.503562+0000","last_became_active":"2024-04-24T00:37:47.426947+0000","last_became_peered":"2024-04-24T00:37:47.426947+0000","last_unstale":"2024-04-24T01:23:09.503562+0000","last_undegraded":"2024-04-24T01:23:09.503562+0000","last_fullsized":"2024-04-24T01:23:09.503562+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:09:53.202558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.138388+0000","last_change":"2024-04-24T00:37:47.431766+0000","last_active":"2024-04-24T01:23:09.138388+0000","last_peered":"2024-04-24T01:23:09.138388+0000","last_clean":"2024-04-24T01:23:09.138388+0000","last_became_active":"2024-04-24T00:37:47.431433+0000","last_became_peered":"2024-04-24T00:37:47.431433+0000","last_unstale":"2024-04-24T01:23:09.138388+0000","last_undegraded":"2024-04-24T01:23:09.138388+0000","last_fullsized":"2024-04-24T01:23:09.138388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:29:01.749521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":805,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.080489+0000","last_change":"2024-04-24T00:37:47.422610+0000","last_active":"2024-04-24T01:23:09.080489+0000","last_peered":"2024-04-24T01:23:09.080489+0000","last_clean":"2024-04-24T01:23:09.080489+0000","last_became_active":"2024-04-24T00:37:47.422197+0000","last_became_peered":"2024-04-24T00:37:47.422197+0000","last_unstale":"2024-04-24T01:23:09.080489+0000","last_undegraded":"2024-04-24T01:23:09.080489+0000","last_fullsized":"2024-04-24T01:23:09.080489+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:02:16.063327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.367103+0000","last_change":"2024-04-24T00:37:47.430799+0000","last_active":"2024-04-24T01:23:09.367103+0000","last_peered":"2024-04-24T01:23:09.367103+0000","last_clean":"2024-04-24T01:23:09.367103+0000","last_became_active":"2024-04-24T00:37:47.430316+0000","last_became_peered":"2024-04-24T00:37:47.430316+0000","last_unstale":"2024-04-24T01:23:09.367103+0000","last_undegraded":"2024-04-24T01:23:09.367103+0000","last_fullsized":"2024-04-24T01:23:09.367103+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:25:37.002437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":937,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.358159+0000","last_change":"2024-04-24T00:37:43.400334+0000","last_active":"2024-04-24T01:23:09.358159+0000","last_peered":"2024-04-24T01:23:09.358159+0000","last_clean":"2024-04-24T01:23:09.358159+0000","last_became_active":"2024-04-24T00:37:43.399665+0000","last_became_peered":"2024-04-24T00:37:43.399665+0000","last_unstale":"2024-04-24T01:23:09.358159+0000","last_undegraded":"2024-04-24T01:23:09.358159+0000","last_fullsized":"2024-04-24T01:23:09.358159+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_clean_scrub_stamp":"2024-04-24T00:37:42.383719+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:35:49.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738923,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1459644,"kb_used_data":4656,"kb_used_omap":142,"kb_used_meta":1454833,"kb_avail":92912196,"statfs":{"total":96636764160,"available":95142088704,"internally_reserved":0,"allocated":4767744,"data_stored":2251843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":146114,"internal_metadata":1489749310},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738921,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1614784,"kb_used_data":5236,"kb_used_omap":124,"kb_used_meta":1609411,"kb_avail":92757056,"statfs":{"total":96636764160,"available":94983225344,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127727,"internal_metadata":1648037137},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738923,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1564996,"kb_used_data":5236,"kb_used_omap":195,"kb_used_meta":1559548,"kb_avail":92806844,"statfs":{"total":96636764160,"available":95034208256,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200294,"internal_metadata":1596977562},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:23:29.445 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-24T01:23:29.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T01:23:29.735 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T01:23:29.735 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T01:23:29.753 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1773,"stamp":"2024-04-24T01:23:29.100246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4639424,"kb_used_data":15128,"kb_used_omap":463,"kb_used_meta":4623792,"kb_avail":278476096,"statfs":{"total":289910292480,"available":285159522304,"internally_reserved":0,"allocated":15491072,"data_stored":7936265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":474135,"internal_metadata":4734764009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003255"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366853+0000","last_change":"2024-04-24T00:37:47.429755+0000","last_active":"2024-04-24T01:23:09.366853+0000","last_peered":"2024-04-24T01:23:09.366853+0000","last_clean":"2024-04-24T01:23:09.366853+0000","last_became_active":"2024-04-24T00:37:47.429213+0000","last_became_peered":"2024-04-24T00:37:47.429213+0000","last_unstale":"2024-04-24T01:23:09.366853+0000","last_undegraded":"2024-04-24T01:23:09.366853+0000","last_fullsized":"2024-04-24T01:23:09.366853+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:34:19.765148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366899+0000","last_change":"2024-04-24T00:37:47.427816+0000","last_active":"2024-04-24T01:23:09.366899+0000","last_peered":"2024-04-24T01:23:09.366899+0000","last_clean":"2024-04-24T01:23:09.366899+0000","last_became_active":"2024-04-24T00:37:47.427410+0000","last_became_peered":"2024-04-24T00:37:47.427410+0000","last_unstale":"2024-04-24T01:23:09.366899+0000","last_undegraded":"2024-04-24T01:23:09.366899+0000","last_fullsized":"2024-04-24T01:23:09.366899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:05:01.512170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366640+0000","last_change":"2024-04-24T00:37:47.428837+0000","last_active":"2024-04-24T01:23:09.366640+0000","last_peered":"2024-04-24T01:23:09.366640+0000","last_clean":"2024-04-24T01:23:09.366640+0000","last_became_active":"2024-04-24T00:37:47.428266+0000","last_became_peered":"2024-04-24T00:37:47.428266+0000","last_unstale":"2024-04-24T01:23:09.366640+0000","last_undegraded":"2024-04-24T01:23:09.366640+0000","last_fullsized":"2024-04-24T01:23:09.366640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:43.968711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.361620+0000","last_change":"2024-04-24T00:37:47.429020+0000","last_active":"2024-04-24T01:23:09.361620+0000","last_peered":"2024-04-24T01:23:09.361620+0000","last_clean":"2024-04-24T01:23:09.361620+0000","last_became_active":"2024-04-24T00:37:47.427471+0000","last_became_peered":"2024-04-24T00:37:47.427471+0000","last_unstale":"2024-04-24T01:23:09.361620+0000","last_undegraded":"2024-04-24T01:23:09.361620+0000","last_fullsized":"2024-04-24T01:23:09.361620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:41.046954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":805,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.503562+0000","last_change":"2024-04-24T00:37:47.427475+0000","last_active":"2024-04-24T01:23:09.503562+0000","last_peered":"2024-04-24T01:23:09.503562+0000","last_clean":"2024-04-24T01:23:09.503562+0000","last_became_active":"2024-04-24T00:37:47.426947+0000","last_became_peered":"2024-04-24T00:37:47.426947+0000","last_unstale":"2024-04-24T01:23:09.503562+0000","last_undegraded":"2024-04-24T01:23:09.503562+0000","last_fullsized":"2024-04-24T01:23:09.503562+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:09:53.202558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.138388+0000","last_change":"2024-04-24T00:37:47.431766+0000","last_active":"2024-04-24T01:23:09.138388+0000","last_peered":"2024-04-24T01:23:09.138388+0000","last_clean":"2024-04-24T01:23:09.138388+0000","last_became_active":"2024-04-24T00:37:47.431433+0000","last_became_peered":"2024-04-24T00:37:47.431433+0000","last_unstale":"2024-04-24T01:23:09.138388+0000","last_undegraded":"2024-04-24T01:23:09.138388+0000","last_fullsized":"2024-04-24T01:23:09.138388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:29:01.749521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":805,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.080489+0000","last_change":"2024-04-24T00:37:47.422610+0000","last_active":"2024-04-24T01:23:09.080489+0000","last_peered":"2024-04-24T01:23:09.080489+0000","last_clean":"2024-04-24T01:23:09.080489+0000","last_became_active":"2024-04-24T00:37:47.422197+0000","last_became_peered":"2024-04-24T00:37:47.422197+0000","last_unstale":"2024-04-24T01:23:09.080489+0000","last_undegraded":"2024-04-24T01:23:09.080489+0000","last_fullsized":"2024-04-24T01:23:09.080489+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:02:16.063327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.367103+0000","last_change":"2024-04-24T00:37:47.430799+0000","last_active":"2024-04-24T01:23:09.367103+0000","last_peered":"2024-04-24T01:23:09.367103+0000","last_clean":"2024-04-24T01:23:09.367103+0000","last_became_active":"2024-04-24T00:37:47.430316+0000","last_became_peered":"2024-04-24T00:37:47.430316+0000","last_unstale":"2024-04-24T01:23:09.367103+0000","last_undegraded":"2024-04-24T01:23:09.367103+0000","last_fullsized":"2024-04-24T01:23:09.367103+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:25:37.002437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":937,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.358159+0000","last_change":"2024-04-24T00:37:43.400334+0000","last_active":"2024-04-24T01:23:09.358159+0000","last_peered":"2024-04-24T01:23:09.358159+0000","last_clean":"2024-04-24T01:23:09.358159+0000","last_became_active":"2024-04-24T00:37:43.399665+0000","last_became_peered":"2024-04-24T00:37:43.399665+0000","last_unstale":"2024-04-24T01:23:09.358159+0000","last_undegraded":"2024-04-24T01:23:09.358159+0000","last_fullsized":"2024-04-24T01:23:09.358159+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_clean_scrub_stamp":"2024-04-24T00:37:42.383719+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:35:49.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738923,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1459644,"kb_used_data":4656,"kb_used_omap":142,"kb_used_meta":1454833,"kb_avail":92912196,"statfs":{"total":96636764160,"available":95142088704,"internally_reserved":0,"allocated":4767744,"data_stored":2251843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":146114,"internal_metadata":1489749310},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738921,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1614784,"kb_used_data":5236,"kb_used_omap":124,"kb_used_meta":1609411,"kb_avail":92757056,"statfs":{"total":96636764160,"available":94983225344,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127727,"internal_metadata":1648037137},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738923,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1564996,"kb_used_data":5236,"kb_used_omap":195,"kb_used_meta":1559548,"kb_avail":92806844,"statfs":{"total":96636764160,"available":95034208256,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200294,"internal_metadata":1596977562},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:23:29.754 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T01:23:30.044 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T01:23:30.044 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":401,"fsid":"f9adc1f2-0792-4cd2-afe7-fbec5c431238","created":"2024-04-24T00:37:30.525473+0000","modified":"2024-04-24T01:23:09.054559+0000","last_up_change":"2024-04-24T00:37:40.143248+0000","last_in_change":"2024-04-24T00:37:31.441935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:37:42.173096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:37:46.090758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e153e69-c5c5-4e91-bada-a7aedb754f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6801","nonce":2330099020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6803","nonce":2330099020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6807","nonce":2330099020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2330099020},{"type":"v1","addr":"172.21.15.134:6805","nonce":2330099020}]},"public_addr":"172.21.15.134:6801/2330099020","cluster_addr":"172.21.15.134:6803/2330099020","heartbeat_back_addr":"172.21.15.134:6807/2330099020","heartbeat_front_addr":"172.21.15.134:6805/2330099020","state":["exists","up"]},{"osd":1,"uuid":"17e7cd39-6768-4e9c-983d-b3cb74060185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6809","nonce":3497594701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6811","nonce":3497594701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6815","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6817","nonce":3497594701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3497594701},{"type":"v1","addr":"172.21.15.134:6814","nonce":3497594701}]},"public_addr":"172.21.15.134:6809/3497594701","cluster_addr":"172.21.15.134:6811/3497594701","heartbeat_back_addr":"172.21.15.134:6817/3497594701","heartbeat_front_addr":"172.21.15.134:6814/3497594701","state":["exists","up"]},{"osd":2,"uuid":"72645446-5208-437b-8acb-67f87d544d82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6813","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6816","nonce":101056980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6819","nonce":101056980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6823","nonce":101056980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":101056980},{"type":"v1","addr":"172.21.15.134:6821","nonce":101056980}]},"public_addr":"172.21.15.134:6816/101056980","cluster_addr":"172.21.15.134:6819/101056980","heartbeat_back_addr":"172.21.15.134:6823/101056980","heartbeat_front_addr":"172.21.15.134:6821/101056980","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.788520+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.877477+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:36.924657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/1988413335":"2024-04-24T02:23:03.676078+0000","172.21.15.134:0/55370313":"2024-04-24T02:21:44.958314+0000","172.21.15.134:0/1921848806":"2024-04-24T02:20:16.867908+0000","172.21.15.134:0/1034011732":"2024-04-24T02:19:35.667899+0000","172.21.15.134:0/2168528873":"2024-04-24T02:18:08.838211+0000","172.21.15.134:0/1496377322":"2024-04-24T02:17:32.498629+0000","172.21.15.134:0/3775657147":"2024-04-24T02:14:37.010853+0000","172.21.15.134:0/3296535612":"2024-04-24T02:06:04.076531+0000","172.21.15.134:0/140073919":"2024-04-24T02:09:40.017719+0000","172.21.15.134:0/2248251789":"2024-04-24T02:06:38.585864+0000","172.21.15.134:0/1184164872":"2024-04-24T02:04:31.989754+0000","172.21.15.134:0/1371095584":"2024-04-24T02:06:19.796351+0000","172.21.15.134:0/4069034790":"2024-04-24T02:11:54.826303+0000","172.21.15.134:0/2954212418":"2024-04-24T02:05:55.761673+0000","172.21.15.134:0/1611882217":"2024-04-24T01:40:11.867461+0000","172.21.15.134:0/668238387":"2024-04-24T02:10:55.344864+0000","172.21.15.134:0/115571028":"2024-04-24T02:04:25.824429+0000","172.21.15.134:0/4116184065":"2024-04-24T02:02:54.490646+0000","172.21.15.134:0/1290462863":"2024-04-24T02:13:44.756558+0000","172.21.15.134:0/1803059490":"2024-04-24T02:06:13.703596+0000","172.21.15.134:0/3417401811":"2024-04-24T02:02:14.217047+0000","172.21.15.134:0/2399984906":"2024-04-24T01:53:28.774894+0000","172.21.15.134:0/258011649":"2024-04-24T02:01:46.410885+0000","172.21.15.134:0/2364983":"2024-04-24T02:12:38.885974+0000","172.21.15.134:0/3897616827":"2024-04-24T02:00:39.776641+0000","172.21.15.134:0/3629133691":"2024-04-24T02:06:49.452366+0000","172.21.15.134:0/2049056039":"2024-04-24T01:41:20.575078+0000","172.21.15.134:0/251362862":"2024-04-24T02:21:01.284816+0000","172.21.15.134:0/2083872363":"2024-04-24T02:12:32.726351+0000","172.21.15.134:0/74924816":"2024-04-24T02:12:24.877618+0000","172.21.15.134:0/1625721287":"2024-04-24T02:06:30.165944+0000","172.21.15.134:0/467725919":"2024-04-24T02:09:50.649232+0000","172.21.15.134:0/2812047840":"2024-04-24T01:47:36.897339+0000","172.21.15.134:0/2194456501":"2024-04-24T02:07:41.600463+0000","172.21.15.134:0/3644645849":"2024-04-24T01:45:15.540277+0000","172.21.15.134:0/79711297":"2024-04-24T02:00:28.362601+0000","172.21.15.134:0/3608706066":"2024-04-24T01:59:58.854379+0000","172.21.15.134:0/3810674242":"2024-04-24T02:04:40.642574+0000","172.21.15.134:0/1990726215":"2024-04-24T01:54:35.179982+0000","172.21.15.134:0/883769784":"2024-04-24T02:06:55.279153+0000","172.21.15.134:0/3383642261":"2024-04-24T02:18:52.447014+0000","172.21.15.134:0/349599712":"2024-04-24T01:49:58.433531+0000","172.21.15.134:0/976316076":"2024-04-24T01:42:47.087768+0000","172.21.15.134:0/1663227597":"2024-04-24T02:09:05.241601+0000","172.21.15.134:0/214630202":"2024-04-24T02:07:59.271275+0000","172.21.15.134:0/3024803658":"2024-04-24T02:07:21.109731+0000","172.21.15.134:0/1170550265":"2024-04-24T02:07:34.894816+0000","172.21.15.134:0/323997629":"2024-04-24T02:08:09.604211+0000","172.21.15.134:0/2030523824":"2024-04-24T02:12:50.592824+0000","172.21.15.134:0/3213982640":"2024-04-24T02:08:25.852551+0000","172.21.15.134:0/1829335069":"2024-04-24T02:08:39.155107+0000","172.21.15.134:0/3619794183":"2024-04-24T02:22:23.351135+0000","172.21.15.134:0/3493801171":"2024-04-24T02:16:55.012182+0000","172.21.15.134:0/2717377373":"2024-04-24T02:05:49.059920+0000","172.21.15.134:0/968551620":"2024-04-24T02:08:50.658287+0000","172.21.15.134:0/2941669734":"2024-04-24T02:09:14.322714+0000","172.21.15.134:0/3764050541":"2024-04-24T02:09:20.570858+0000","172.21.15.134:0/2098510276":"2024-04-24T02:10:32.289026+0000","172.21.15.134:0/3735116525":"2024-04-24T02:14:25.180169+0000","172.21.15.134:0/2754305220":"2024-04-24T01:57:35.592020+0000","172.21.15.134:0/889751334":"2024-04-24T02:10:45.923447+0000","172.21.15.134:0/292367832":"2024-04-24T01:51:28.148855+0000","172.21.15.134:0/2124668253":"2024-04-24T01:52:32.907656+0000","172.21.15.134:0/790511785":"2024-04-24T02:02:19.854348+0000","172.21.15.134:0/727486921":"2024-04-24T02:14:13.037858+0000","172.21.15.134:0/738794570":"2024-04-24T02:12:06.751864+0000","172.21.15.134:0/1361002848":"2024-04-24T02:12:14.592064+0000","172.21.15.134:0/3065630456":"2024-04-24T02:07:10.655650+0000","172.21.15.134:0/1945067181":"2024-04-24T02:13:05.547476+0000","172.21.15.134:0/729178199":"2024-04-24T02:13:59.977343+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T01:23:31.064 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-04-24T01:23:31.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T01:23:31.219 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-24T01:23:31.219 INFO:teuthology.orchestra.run.smithi134.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T01:23:31.219 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-24T01:23:31.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-04-24T01:23:31.520 INFO:teuthology.orchestra.run.smithi134.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T01:23:31.538 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-04-24T01:23:31.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T01:23:31.673 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-24T01:23:31.673 INFO:teuthology.orchestra.run.smithi134.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T01:23:31.673 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-24T01:23:31.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-04-24T01:23:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T01:23:31.978 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-04-24T01:23:31.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T01:23:32.120 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-24T01:23:32.121 INFO:teuthology.orchestra.run.smithi134.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T01:23:32.121 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-24T01:23:32.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-04-24T01:23:32.396 INFO:teuthology.orchestra.run.smithi134.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T01:23:32.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T01:23:32.693 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T01:23:32.693 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T01:23:32.710 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1774,"stamp":"2024-04-24T01:23:31.100751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4639424,"kb_used_data":15128,"kb_used_omap":463,"kb_used_meta":4623792,"kb_avail":278476096,"statfs":{"total":289910292480,"available":285159522304,"internally_reserved":0,"allocated":15491072,"data_stored":7936265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":474135,"internal_metadata":4734764009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003168"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366853+0000","last_change":"2024-04-24T00:37:47.429755+0000","last_active":"2024-04-24T01:23:09.366853+0000","last_peered":"2024-04-24T01:23:09.366853+0000","last_clean":"2024-04-24T01:23:09.366853+0000","last_became_active":"2024-04-24T00:37:47.429213+0000","last_became_peered":"2024-04-24T00:37:47.429213+0000","last_unstale":"2024-04-24T01:23:09.366853+0000","last_undegraded":"2024-04-24T01:23:09.366853+0000","last_fullsized":"2024-04-24T01:23:09.366853+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:34:19.765148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366899+0000","last_change":"2024-04-24T00:37:47.427816+0000","last_active":"2024-04-24T01:23:09.366899+0000","last_peered":"2024-04-24T01:23:09.366899+0000","last_clean":"2024-04-24T01:23:09.366899+0000","last_became_active":"2024-04-24T00:37:47.427410+0000","last_became_peered":"2024-04-24T00:37:47.427410+0000","last_unstale":"2024-04-24T01:23:09.366899+0000","last_undegraded":"2024-04-24T01:23:09.366899+0000","last_fullsized":"2024-04-24T01:23:09.366899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:05:01.512170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.366640+0000","last_change":"2024-04-24T00:37:47.428837+0000","last_active":"2024-04-24T01:23:09.366640+0000","last_peered":"2024-04-24T01:23:09.366640+0000","last_clean":"2024-04-24T01:23:09.366640+0000","last_became_active":"2024-04-24T00:37:47.428266+0000","last_became_peered":"2024-04-24T00:37:47.428266+0000","last_unstale":"2024-04-24T01:23:09.366640+0000","last_undegraded":"2024-04-24T01:23:09.366640+0000","last_fullsized":"2024-04-24T01:23:09.366640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:43.968711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.361620+0000","last_change":"2024-04-24T00:37:47.429020+0000","last_active":"2024-04-24T01:23:09.361620+0000","last_peered":"2024-04-24T01:23:09.361620+0000","last_clean":"2024-04-24T01:23:09.361620+0000","last_became_active":"2024-04-24T00:37:47.427471+0000","last_became_peered":"2024-04-24T00:37:47.427471+0000","last_unstale":"2024-04-24T01:23:09.361620+0000","last_undegraded":"2024-04-24T01:23:09.361620+0000","last_fullsized":"2024-04-24T01:23:09.361620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:41.046954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":805,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.503562+0000","last_change":"2024-04-24T00:37:47.427475+0000","last_active":"2024-04-24T01:23:09.503562+0000","last_peered":"2024-04-24T01:23:09.503562+0000","last_clean":"2024-04-24T01:23:09.503562+0000","last_became_active":"2024-04-24T00:37:47.426947+0000","last_became_peered":"2024-04-24T00:37:47.426947+0000","last_unstale":"2024-04-24T01:23:09.503562+0000","last_undegraded":"2024-04-24T01:23:09.503562+0000","last_fullsized":"2024-04-24T01:23:09.503562+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:09:53.202558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.138388+0000","last_change":"2024-04-24T00:37:47.431766+0000","last_active":"2024-04-24T01:23:09.138388+0000","last_peered":"2024-04-24T01:23:09.138388+0000","last_clean":"2024-04-24T01:23:09.138388+0000","last_became_active":"2024-04-24T00:37:47.431433+0000","last_became_peered":"2024-04-24T00:37:47.431433+0000","last_unstale":"2024-04-24T01:23:09.138388+0000","last_undegraded":"2024-04-24T01:23:09.138388+0000","last_fullsized":"2024-04-24T01:23:09.138388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:29:01.749521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":805,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.080489+0000","last_change":"2024-04-24T00:37:47.422610+0000","last_active":"2024-04-24T01:23:09.080489+0000","last_peered":"2024-04-24T01:23:09.080489+0000","last_clean":"2024-04-24T01:23:09.080489+0000","last_became_active":"2024-04-24T00:37:47.422197+0000","last_became_peered":"2024-04-24T00:37:47.422197+0000","last_unstale":"2024-04-24T01:23:09.080489+0000","last_undegraded":"2024-04-24T01:23:09.080489+0000","last_fullsized":"2024-04-24T01:23:09.080489+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:02:16.063327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":804,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.367103+0000","last_change":"2024-04-24T00:37:47.430799+0000","last_active":"2024-04-24T01:23:09.367103+0000","last_peered":"2024-04-24T01:23:09.367103+0000","last_clean":"2024-04-24T01:23:09.367103+0000","last_became_active":"2024-04-24T00:37:47.430316+0000","last_became_peered":"2024-04-24T00:37:47.430316+0000","last_unstale":"2024-04-24T01:23:09.367103+0000","last_undegraded":"2024-04-24T01:23:09.367103+0000","last_fullsized":"2024-04-24T01:23:09.367103+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:46.402139+0000","last_clean_scrub_stamp":"2024-04-24T00:37:46.402139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:25:37.002437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":937,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:09.358159+0000","last_change":"2024-04-24T00:37:43.400334+0000","last_active":"2024-04-24T01:23:09.358159+0000","last_peered":"2024-04-24T01:23:09.358159+0000","last_clean":"2024-04-24T01:23:09.358159+0000","last_became_active":"2024-04-24T00:37:43.399665+0000","last_became_peered":"2024-04-24T00:37:43.399665+0000","last_unstale":"2024-04-24T01:23:09.358159+0000","last_undegraded":"2024-04-24T01:23:09.358159+0000","last_fullsized":"2024-04-24T01:23:09.358159+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:42.383719+0000","last_clean_scrub_stamp":"2024-04-24T00:37:42.383719+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:35:49.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738924,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1459644,"kb_used_data":4656,"kb_used_omap":142,"kb_used_meta":1454833,"kb_avail":92912196,"statfs":{"total":96636764160,"available":95142088704,"internally_reserved":0,"allocated":4767744,"data_stored":2251843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":146114,"internal_metadata":1489749310},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738922,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1614784,"kb_used_data":5236,"kb_used_omap":124,"kb_used_meta":1609411,"kb_avail":92757056,"statfs":{"total":96636764160,"available":94983225344,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127727,"internal_metadata":1648037137},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738923,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1564996,"kb_used_data":5236,"kb_used_omap":195,"kb_used_meta":1559548,"kb_avail":92806844,"statfs":{"total":96636764160,"available":95034208256,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200294,"internal_metadata":1596977562},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:23:32.714 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-24T00:37:46.402139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=46, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=23, tm_sec=30, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:23:32.714 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T00:37:46.402139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=46, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=23, tm_sec=30, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:23:32.714 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-24T00:37:46.402139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=46, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=23, tm_sec=30, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:23:32.714 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-24T00:37:46.402139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=46, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=23, tm_sec=30, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:23:32.714 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-24T00:37:46.402139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=46, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=23, tm_sec=30, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:23:32.715 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-24T00:37:46.402139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=46, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=23, tm_sec=30, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:23:32.715 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T00:37:46.402139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=46, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=23, tm_sec=30, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:23:32.715 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-24T00:37:46.402139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=46, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=23, tm_sec=30, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:23:32.715 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T00:37:42.383719+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=23, tm_sec=30, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:23:32.715 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T01:23:52.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T01:23:53.017 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T01:23:53.018 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T01:23:53.036 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1784,"stamp":"2024-04-24T01:23:51.105845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4639408,"kb_used_data":15128,"kb_used_omap":463,"kb_used_meta":4623792,"kb_avail":278476112,"statfs":{"total":289910292480,"available":285159538688,"internally_reserved":0,"allocated":15491072,"data_stored":7936265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":474135,"internal_metadata":4734764009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003062"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":813,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:36.409340+0000","last_change":"2024-04-24T01:23:36.409363+0000","last_active":"2024-04-24T01:23:36.409340+0000","last_peered":"2024-04-24T01:23:36.409340+0000","last_clean":"2024-04-24T01:23:36.409340+0000","last_became_active":"2024-04-24T00:37:47.429213+0000","last_became_peered":"2024-04-24T00:37:47.429213+0000","last_unstale":"2024-04-24T01:23:36.409340+0000","last_undegraded":"2024-04-24T01:23:36.409340+0000","last_fullsized":"2024-04-24T01:23:36.409340+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:23:36.409223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:23:36.409223+0000","last_clean_scrub_stamp":"2024-04-24T01:23:36.409223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:57:21.839922+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":813,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:33.204168+0000","last_change":"2024-04-24T01:23:33.204191+0000","last_active":"2024-04-24T01:23:33.204168+0000","last_peered":"2024-04-24T01:23:33.204168+0000","last_clean":"2024-04-24T01:23:33.204168+0000","last_became_active":"2024-04-24T00:37:47.427410+0000","last_became_peered":"2024-04-24T00:37:47.427410+0000","last_unstale":"2024-04-24T01:23:33.204168+0000","last_undegraded":"2024-04-24T01:23:33.204168+0000","last_fullsized":"2024-04-24T01:23:33.204168+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:23:33.204073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:23:33.204073+0000","last_clean_scrub_stamp":"2024-04-24T01:23:33.204073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:29:01.631383+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":813,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:34.319435+0000","last_change":"2024-04-24T01:23:34.319456+0000","last_active":"2024-04-24T01:23:34.319435+0000","last_peered":"2024-04-24T01:23:34.319435+0000","last_clean":"2024-04-24T01:23:34.319435+0000","last_became_active":"2024-04-24T00:37:47.428266+0000","last_became_peered":"2024-04-24T00:37:47.428266+0000","last_unstale":"2024-04-24T01:23:34.319435+0000","last_undegraded":"2024-04-24T01:23:34.319435+0000","last_fullsized":"2024-04-24T01:23:34.319435+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:23:34.319324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:23:34.319324+0000","last_clean_scrub_stamp":"2024-04-24T01:23:34.319324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:01:48.682296+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":814,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:35.114737+0000","last_change":"2024-04-24T01:23:35.114737+0000","last_active":"2024-04-24T01:23:35.114737+0000","last_peered":"2024-04-24T01:23:35.114737+0000","last_clean":"2024-04-24T01:23:35.114737+0000","last_became_active":"2024-04-24T00:37:47.427471+0000","last_became_peered":"2024-04-24T00:37:47.427471+0000","last_unstale":"2024-04-24T01:23:35.114737+0000","last_undegraded":"2024-04-24T01:23:35.114737+0000","last_fullsized":"2024-04-24T01:23:35.114737+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:23:35.114479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:23:35.114479+0000","last_clean_scrub_stamp":"2024-04-24T01:23:35.114479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:33.058914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":815,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:31.623797+0000","last_change":"2024-04-24T01:23:31.623797+0000","last_active":"2024-04-24T01:23:31.623797+0000","last_peered":"2024-04-24T01:23:31.623797+0000","last_clean":"2024-04-24T01:23:31.623797+0000","last_became_active":"2024-04-24T00:37:47.426947+0000","last_became_peered":"2024-04-24T00:37:47.426947+0000","last_unstale":"2024-04-24T01:23:31.623797+0000","last_undegraded":"2024-04-24T01:23:31.623797+0000","last_fullsized":"2024-04-24T01:23:31.623797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:23:31.623630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:23:31.623630+0000","last_clean_scrub_stamp":"2024-04-24T01:23:31.623630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:53:35.785537+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":814,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:34.340873+0000","last_change":"2024-04-24T01:23:34.340873+0000","last_active":"2024-04-24T01:23:34.340873+0000","last_peered":"2024-04-24T01:23:34.340873+0000","last_clean":"2024-04-24T01:23:34.340873+0000","last_became_active":"2024-04-24T00:37:47.431433+0000","last_became_peered":"2024-04-24T00:37:47.431433+0000","last_unstale":"2024-04-24T01:23:34.340873+0000","last_undegraded":"2024-04-24T01:23:34.340873+0000","last_fullsized":"2024-04-24T01:23:34.340873+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:23:34.340649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:23:34.340649+0000","last_clean_scrub_stamp":"2024-04-24T01:23:34.340649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:18:08.675307+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":814,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:33.230332+0000","last_change":"2024-04-24T01:23:33.230354+0000","last_active":"2024-04-24T01:23:33.230332+0000","last_peered":"2024-04-24T01:23:33.230332+0000","last_clean":"2024-04-24T01:23:33.230332+0000","last_became_active":"2024-04-24T00:37:47.422197+0000","last_became_peered":"2024-04-24T00:37:47.422197+0000","last_unstale":"2024-04-24T01:23:33.230332+0000","last_undegraded":"2024-04-24T01:23:33.230332+0000","last_fullsized":"2024-04-24T01:23:33.230332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:23:33.230249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:23:33.230249+0000","last_clean_scrub_stamp":"2024-04-24T01:23:33.230249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:16:30.724321+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":814,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:37.413889+0000","last_change":"2024-04-24T01:23:37.413889+0000","last_active":"2024-04-24T01:23:37.413889+0000","last_peered":"2024-04-24T01:23:37.413889+0000","last_clean":"2024-04-24T01:23:37.413889+0000","last_became_active":"2024-04-24T00:37:47.430316+0000","last_became_peered":"2024-04-24T00:37:47.430316+0000","last_unstale":"2024-04-24T01:23:37.413889+0000","last_undegraded":"2024-04-24T01:23:37.413889+0000","last_fullsized":"2024-04-24T01:23:37.413889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:23:37.413679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:23:37.413679+0000","last_clean_scrub_stamp":"2024-04-24T01:23:37.413679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:50.945455+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"401'99","reported_seq":948,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-24T01:23:32.063848+0000","last_change":"2024-04-24T01:23:32.064003+0000","last_active":"2024-04-24T01:23:32.063848+0000","last_peered":"2024-04-24T01:23:32.063848+0000","last_clean":"2024-04-24T01:23:32.063848+0000","last_became_active":"2024-04-24T00:37:43.399665+0000","last_became_peered":"2024-04-24T00:37:43.399665+0000","last_unstale":"2024-04-24T01:23:32.063848+0000","last_undegraded":"2024-04-24T01:23:32.063848+0000","last_fullsized":"2024-04-24T01:23:32.063848+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'99","last_scrub_stamp":"2024-04-24T01:23:32.063626+0000","last_deep_scrub":"401'99","last_deep_scrub_stamp":"2024-04-24T01:23:32.063626+0000","last_clean_scrub_stamp":"2024-04-24T01:23:32.063626+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:41.427178+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738928,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1459640,"kb_used_data":4656,"kb_used_omap":142,"kb_used_meta":1454833,"kb_avail":92912200,"statfs":{"total":96636764160,"available":95142092800,"internally_reserved":0,"allocated":4767744,"data_stored":2251843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":146114,"internal_metadata":1489749310},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738926,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1614780,"kb_used_data":5236,"kb_used_omap":124,"kb_used_meta":1609411,"kb_avail":92757060,"statfs":{"total":96636764160,"available":94983229440,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127727,"internal_metadata":1648037137},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738927,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1564988,"kb_used_data":5236,"kb_used_omap":195,"kb_used_meta":1559548,"kb_avail":92806852,"statfs":{"total":96636764160,"available":95034216448,"internally_reserved":0,"allocated":5361664,"data_stored":2842211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200294,"internal_metadata":1596977562},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:23:53.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-04-24T01:23:53.355 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T01:23:53.356 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T01:23:53.356 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-04-24T01:23:53.356 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:23:56.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-04-24T01:23:59.457 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-04-24T01:23:59.458 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-04-24T01:23:59.458 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:24:01.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-04-24T01:24:01.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-04-24T01:24:05.560 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-04-24T01:24:05.560 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-04-24T01:24:05.561 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:24:07.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2024-04-24T01:24:07.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-04-24T01:24:11.663 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-04-24T01:24:11.663 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T01:24:11.663 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-04-24T01:24:11.664 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:24:11.712 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-04-24T01:24:11.712 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T01:24:11.713 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-04-24T01:24:11.713 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:24:11.740 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-04-24T01:24:11.740 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T01:24:11.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-24T01:24:11.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T01:24:11.867 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-04-24T01:24:12.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2024-04-24T01:24:12.243 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T01:24:12.243 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-04-24T01:24:12.588 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T01:24:12.588 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-04-24T01:24:12.932 INFO:tasks.ceph:Archiving mon data... 2024-04-24T01:24:12.933 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535/data/cluster2.mon.a.tgz 2024-04-24T01:24:12.934 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-04-24T01:24:12.938 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.ENeySjJuhW 2024-04-24T01:24:12.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.ENeySjJuhW 2024-04-24T01:24:13.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.ENeySjJuhW 2024-04-24T01:24:13.169 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.ENeySjJuhW is 423KB 2024-04-24T01:24:13.228 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.ENeySjJuhW 2024-04-24T01:24:13.233 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T01:24:13.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T01:24:13.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T01:24:13.460 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T01:24:13.474 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-24T01:24:13.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T01:24:14.003 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T01:24:14.004 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-24T01:24:14.041 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1749,"stamp":"2024-04-24T01:24:13.667092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3086140,"kb_used_data":9296,"kb_used_omap":501,"kb_used_meta":3076298,"kb_avail":280029380,"statfs":{"total":289910292480,"available":286750085120,"internally_reserved":0,"allocated":9519104,"data_stored":5613809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":513191,"internal_metadata":3150130009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.220676+0000","last_change":"2024-04-24T00:37:10.788095+0000","last_active":"2024-04-24T01:23:08.220676+0000","last_peered":"2024-04-24T01:23:08.220676+0000","last_clean":"2024-04-24T01:23:08.220676+0000","last_became_active":"2024-04-24T00:37:10.787769+0000","last_became_peered":"2024-04-24T00:37:10.787769+0000","last_unstale":"2024-04-24T01:23:08.220676+0000","last_undegraded":"2024-04-24T01:23:08.220676+0000","last_fullsized":"2024-04-24T01:23:08.220676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:23:01.042116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.223591+0000","last_change":"2024-04-24T00:37:10.788073+0000","last_active":"2024-04-24T01:23:08.223591+0000","last_peered":"2024-04-24T01:23:08.223591+0000","last_clean":"2024-04-24T01:23:08.223591+0000","last_became_active":"2024-04-24T00:37:10.787658+0000","last_became_peered":"2024-04-24T00:37:10.787658+0000","last_unstale":"2024-04-24T01:23:08.223591+0000","last_undegraded":"2024-04-24T01:23:08.223591+0000","last_fullsized":"2024-04-24T01:23:08.223591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:32:27.147352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.222009+0000","last_change":"2024-04-24T00:37:10.785452+0000","last_active":"2024-04-24T01:23:08.222009+0000","last_peered":"2024-04-24T01:23:08.222009+0000","last_clean":"2024-04-24T01:23:08.222009+0000","last_became_active":"2024-04-24T00:37:10.785100+0000","last_became_peered":"2024-04-24T00:37:10.785100+0000","last_unstale":"2024-04-24T01:23:08.222009+0000","last_undegraded":"2024-04-24T01:23:08.222009+0000","last_fullsized":"2024-04-24T01:23:08.222009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:59.597565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.219841+0000","last_change":"2024-04-24T00:37:10.787976+0000","last_active":"2024-04-24T01:23:08.219841+0000","last_peered":"2024-04-24T01:23:08.219841+0000","last_clean":"2024-04-24T01:23:08.219841+0000","last_became_active":"2024-04-24T00:37:10.787561+0000","last_became_peered":"2024-04-24T00:37:10.787561+0000","last_unstale":"2024-04-24T01:23:08.219841+0000","last_undegraded":"2024-04-24T01:23:08.219841+0000","last_fullsized":"2024-04-24T01:23:08.219841+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:30:08.931603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.514147+0000","last_change":"2024-04-24T00:37:10.786163+0000","last_active":"2024-04-24T01:23:08.514147+0000","last_peered":"2024-04-24T01:23:08.514147+0000","last_clean":"2024-04-24T01:23:08.514147+0000","last_became_active":"2024-04-24T00:37:10.785718+0000","last_became_peered":"2024-04-24T00:37:10.785718+0000","last_unstale":"2024-04-24T01:23:08.514147+0000","last_undegraded":"2024-04-24T01:23:08.514147+0000","last_fullsized":"2024-04-24T01:23:08.514147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:57.680592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.517744+0000","last_change":"2024-04-24T00:37:10.786864+0000","last_active":"2024-04-24T01:23:08.517744+0000","last_peered":"2024-04-24T01:23:08.517744+0000","last_clean":"2024-04-24T01:23:08.517744+0000","last_became_active":"2024-04-24T00:37:10.785638+0000","last_became_peered":"2024-04-24T00:37:10.785638+0000","last_unstale":"2024-04-24T01:23:08.517744+0000","last_undegraded":"2024-04-24T01:23:08.517744+0000","last_fullsized":"2024-04-24T01:23:08.517744+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:14.839760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":681,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.513030+0000","last_change":"2024-04-24T00:37:10.786316+0000","last_active":"2024-04-24T01:23:08.513030+0000","last_peered":"2024-04-24T01:23:08.513030+0000","last_clean":"2024-04-24T01:23:08.513030+0000","last_became_active":"2024-04-24T00:37:10.785901+0000","last_became_peered":"2024-04-24T00:37:10.785901+0000","last_unstale":"2024-04-24T01:23:08.513030+0000","last_undegraded":"2024-04-24T01:23:08.513030+0000","last_fullsized":"2024-04-24T01:23:08.513030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:45:47.984157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.220730+0000","last_change":"2024-04-24T00:37:10.785726+0000","last_active":"2024-04-24T01:23:08.220730+0000","last_peered":"2024-04-24T01:23:08.220730+0000","last_clean":"2024-04-24T01:23:08.220730+0000","last_became_active":"2024-04-24T00:37:10.785333+0000","last_became_peered":"2024-04-24T00:37:10.785333+0000","last_unstale":"2024-04-24T01:23:08.220730+0000","last_undegraded":"2024-04-24T01:23:08.220730+0000","last_fullsized":"2024-04-24T01:23:08.220730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:24:20.306473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":814,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.221748+0000","last_change":"2024-04-24T00:37:05.801930+0000","last_active":"2024-04-24T01:23:08.221748+0000","last_peered":"2024-04-24T01:23:08.221748+0000","last_clean":"2024-04-24T01:23:08.221748+0000","last_became_active":"2024-04-24T00:37:05.799306+0000","last_became_peered":"2024-04-24T00:37:05.799306+0000","last_unstale":"2024-04-24T01:23:08.221748+0000","last_undegraded":"2024-04-24T01:23:08.221748+0000","last_fullsized":"2024-04-24T01:23:08.221748+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_clean_scrub_stamp":"2024-04-24T00:37:04.744801+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:37:53.503175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738938,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935784,"kb_used_data":2712,"kb_used_omap":148,"kb_used_meta":932907,"kb_avail":93436056,"statfs":{"total":96636764160,"available":95678521344,"internally_reserved":0,"allocated":2777088,"data_stored":1477691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":152420,"internal_metadata":955296924},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738939,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1482920,"kb_used_data":3292,"kb_used_omap":131,"kb_used_meta":1479484,"kb_avail":92888920,"statfs":{"total":96636764160,"available":95118254080,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134892,"internal_metadata":1514991892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738940,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667436,"kb_used_data":3292,"kb_used_omap":220,"kb_used_meta":663907,"kb_avail":93704404,"statfs":{"total":96636764160,"available":95953309696,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225879,"internal_metadata":679841193},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:24:14.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T01:24:14.331 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T01:24:14.331 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-24T01:24:14.348 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1749,"stamp":"2024-04-24T01:24:13.667092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3086140,"kb_used_data":9296,"kb_used_omap":501,"kb_used_meta":3076298,"kb_avail":280029380,"statfs":{"total":289910292480,"available":286750085120,"internally_reserved":0,"allocated":9519104,"data_stored":5613809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":513191,"internal_metadata":3150130009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.220676+0000","last_change":"2024-04-24T00:37:10.788095+0000","last_active":"2024-04-24T01:23:08.220676+0000","last_peered":"2024-04-24T01:23:08.220676+0000","last_clean":"2024-04-24T01:23:08.220676+0000","last_became_active":"2024-04-24T00:37:10.787769+0000","last_became_peered":"2024-04-24T00:37:10.787769+0000","last_unstale":"2024-04-24T01:23:08.220676+0000","last_undegraded":"2024-04-24T01:23:08.220676+0000","last_fullsized":"2024-04-24T01:23:08.220676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:23:01.042116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.223591+0000","last_change":"2024-04-24T00:37:10.788073+0000","last_active":"2024-04-24T01:23:08.223591+0000","last_peered":"2024-04-24T01:23:08.223591+0000","last_clean":"2024-04-24T01:23:08.223591+0000","last_became_active":"2024-04-24T00:37:10.787658+0000","last_became_peered":"2024-04-24T00:37:10.787658+0000","last_unstale":"2024-04-24T01:23:08.223591+0000","last_undegraded":"2024-04-24T01:23:08.223591+0000","last_fullsized":"2024-04-24T01:23:08.223591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:32:27.147352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.222009+0000","last_change":"2024-04-24T00:37:10.785452+0000","last_active":"2024-04-24T01:23:08.222009+0000","last_peered":"2024-04-24T01:23:08.222009+0000","last_clean":"2024-04-24T01:23:08.222009+0000","last_became_active":"2024-04-24T00:37:10.785100+0000","last_became_peered":"2024-04-24T00:37:10.785100+0000","last_unstale":"2024-04-24T01:23:08.222009+0000","last_undegraded":"2024-04-24T01:23:08.222009+0000","last_fullsized":"2024-04-24T01:23:08.222009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:59.597565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.219841+0000","last_change":"2024-04-24T00:37:10.787976+0000","last_active":"2024-04-24T01:23:08.219841+0000","last_peered":"2024-04-24T01:23:08.219841+0000","last_clean":"2024-04-24T01:23:08.219841+0000","last_became_active":"2024-04-24T00:37:10.787561+0000","last_became_peered":"2024-04-24T00:37:10.787561+0000","last_unstale":"2024-04-24T01:23:08.219841+0000","last_undegraded":"2024-04-24T01:23:08.219841+0000","last_fullsized":"2024-04-24T01:23:08.219841+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:30:08.931603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.514147+0000","last_change":"2024-04-24T00:37:10.786163+0000","last_active":"2024-04-24T01:23:08.514147+0000","last_peered":"2024-04-24T01:23:08.514147+0000","last_clean":"2024-04-24T01:23:08.514147+0000","last_became_active":"2024-04-24T00:37:10.785718+0000","last_became_peered":"2024-04-24T00:37:10.785718+0000","last_unstale":"2024-04-24T01:23:08.514147+0000","last_undegraded":"2024-04-24T01:23:08.514147+0000","last_fullsized":"2024-04-24T01:23:08.514147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:57.680592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.517744+0000","last_change":"2024-04-24T00:37:10.786864+0000","last_active":"2024-04-24T01:23:08.517744+0000","last_peered":"2024-04-24T01:23:08.517744+0000","last_clean":"2024-04-24T01:23:08.517744+0000","last_became_active":"2024-04-24T00:37:10.785638+0000","last_became_peered":"2024-04-24T00:37:10.785638+0000","last_unstale":"2024-04-24T01:23:08.517744+0000","last_undegraded":"2024-04-24T01:23:08.517744+0000","last_fullsized":"2024-04-24T01:23:08.517744+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:14.839760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":681,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.513030+0000","last_change":"2024-04-24T00:37:10.786316+0000","last_active":"2024-04-24T01:23:08.513030+0000","last_peered":"2024-04-24T01:23:08.513030+0000","last_clean":"2024-04-24T01:23:08.513030+0000","last_became_active":"2024-04-24T00:37:10.785901+0000","last_became_peered":"2024-04-24T00:37:10.785901+0000","last_unstale":"2024-04-24T01:23:08.513030+0000","last_undegraded":"2024-04-24T01:23:08.513030+0000","last_fullsized":"2024-04-24T01:23:08.513030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:45:47.984157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.220730+0000","last_change":"2024-04-24T00:37:10.785726+0000","last_active":"2024-04-24T01:23:08.220730+0000","last_peered":"2024-04-24T01:23:08.220730+0000","last_clean":"2024-04-24T01:23:08.220730+0000","last_became_active":"2024-04-24T00:37:10.785333+0000","last_became_peered":"2024-04-24T00:37:10.785333+0000","last_unstale":"2024-04-24T01:23:08.220730+0000","last_undegraded":"2024-04-24T01:23:08.220730+0000","last_fullsized":"2024-04-24T01:23:08.220730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:24:20.306473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":814,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.221748+0000","last_change":"2024-04-24T00:37:05.801930+0000","last_active":"2024-04-24T01:23:08.221748+0000","last_peered":"2024-04-24T01:23:08.221748+0000","last_clean":"2024-04-24T01:23:08.221748+0000","last_became_active":"2024-04-24T00:37:05.799306+0000","last_became_peered":"2024-04-24T00:37:05.799306+0000","last_unstale":"2024-04-24T01:23:08.221748+0000","last_undegraded":"2024-04-24T01:23:08.221748+0000","last_fullsized":"2024-04-24T01:23:08.221748+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_clean_scrub_stamp":"2024-04-24T00:37:04.744801+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:37:53.503175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738938,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935784,"kb_used_data":2712,"kb_used_omap":148,"kb_used_meta":932907,"kb_avail":93436056,"statfs":{"total":96636764160,"available":95678521344,"internally_reserved":0,"allocated":2777088,"data_stored":1477691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":152420,"internal_metadata":955296924},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738939,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1482920,"kb_used_data":3292,"kb_used_omap":131,"kb_used_meta":1479484,"kb_avail":92888920,"statfs":{"total":96636764160,"available":95118254080,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134892,"internal_metadata":1514991892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738940,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667436,"kb_used_data":3292,"kb_used_omap":220,"kb_used_meta":663907,"kb_avail":93704404,"statfs":{"total":96636764160,"available":95953309696,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225879,"internal_metadata":679841193},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:24:14.349 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-24T01:24:14.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T01:24:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T01:24:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-24T01:24:14.654 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1749,"stamp":"2024-04-24T01:24:13.667092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3086140,"kb_used_data":9296,"kb_used_omap":501,"kb_used_meta":3076298,"kb_avail":280029380,"statfs":{"total":289910292480,"available":286750085120,"internally_reserved":0,"allocated":9519104,"data_stored":5613809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":513191,"internal_metadata":3150130009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.220676+0000","last_change":"2024-04-24T00:37:10.788095+0000","last_active":"2024-04-24T01:23:08.220676+0000","last_peered":"2024-04-24T01:23:08.220676+0000","last_clean":"2024-04-24T01:23:08.220676+0000","last_became_active":"2024-04-24T00:37:10.787769+0000","last_became_peered":"2024-04-24T00:37:10.787769+0000","last_unstale":"2024-04-24T01:23:08.220676+0000","last_undegraded":"2024-04-24T01:23:08.220676+0000","last_fullsized":"2024-04-24T01:23:08.220676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:23:01.042116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.223591+0000","last_change":"2024-04-24T00:37:10.788073+0000","last_active":"2024-04-24T01:23:08.223591+0000","last_peered":"2024-04-24T01:23:08.223591+0000","last_clean":"2024-04-24T01:23:08.223591+0000","last_became_active":"2024-04-24T00:37:10.787658+0000","last_became_peered":"2024-04-24T00:37:10.787658+0000","last_unstale":"2024-04-24T01:23:08.223591+0000","last_undegraded":"2024-04-24T01:23:08.223591+0000","last_fullsized":"2024-04-24T01:23:08.223591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:32:27.147352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.222009+0000","last_change":"2024-04-24T00:37:10.785452+0000","last_active":"2024-04-24T01:23:08.222009+0000","last_peered":"2024-04-24T01:23:08.222009+0000","last_clean":"2024-04-24T01:23:08.222009+0000","last_became_active":"2024-04-24T00:37:10.785100+0000","last_became_peered":"2024-04-24T00:37:10.785100+0000","last_unstale":"2024-04-24T01:23:08.222009+0000","last_undegraded":"2024-04-24T01:23:08.222009+0000","last_fullsized":"2024-04-24T01:23:08.222009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:59.597565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.219841+0000","last_change":"2024-04-24T00:37:10.787976+0000","last_active":"2024-04-24T01:23:08.219841+0000","last_peered":"2024-04-24T01:23:08.219841+0000","last_clean":"2024-04-24T01:23:08.219841+0000","last_became_active":"2024-04-24T00:37:10.787561+0000","last_became_peered":"2024-04-24T00:37:10.787561+0000","last_unstale":"2024-04-24T01:23:08.219841+0000","last_undegraded":"2024-04-24T01:23:08.219841+0000","last_fullsized":"2024-04-24T01:23:08.219841+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:30:08.931603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.514147+0000","last_change":"2024-04-24T00:37:10.786163+0000","last_active":"2024-04-24T01:23:08.514147+0000","last_peered":"2024-04-24T01:23:08.514147+0000","last_clean":"2024-04-24T01:23:08.514147+0000","last_became_active":"2024-04-24T00:37:10.785718+0000","last_became_peered":"2024-04-24T00:37:10.785718+0000","last_unstale":"2024-04-24T01:23:08.514147+0000","last_undegraded":"2024-04-24T01:23:08.514147+0000","last_fullsized":"2024-04-24T01:23:08.514147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:57.680592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.517744+0000","last_change":"2024-04-24T00:37:10.786864+0000","last_active":"2024-04-24T01:23:08.517744+0000","last_peered":"2024-04-24T01:23:08.517744+0000","last_clean":"2024-04-24T01:23:08.517744+0000","last_became_active":"2024-04-24T00:37:10.785638+0000","last_became_peered":"2024-04-24T00:37:10.785638+0000","last_unstale":"2024-04-24T01:23:08.517744+0000","last_undegraded":"2024-04-24T01:23:08.517744+0000","last_fullsized":"2024-04-24T01:23:08.517744+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:14.839760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":681,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.513030+0000","last_change":"2024-04-24T00:37:10.786316+0000","last_active":"2024-04-24T01:23:08.513030+0000","last_peered":"2024-04-24T01:23:08.513030+0000","last_clean":"2024-04-24T01:23:08.513030+0000","last_became_active":"2024-04-24T00:37:10.785901+0000","last_became_peered":"2024-04-24T00:37:10.785901+0000","last_unstale":"2024-04-24T01:23:08.513030+0000","last_undegraded":"2024-04-24T01:23:08.513030+0000","last_fullsized":"2024-04-24T01:23:08.513030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:45:47.984157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.220730+0000","last_change":"2024-04-24T00:37:10.785726+0000","last_active":"2024-04-24T01:23:08.220730+0000","last_peered":"2024-04-24T01:23:08.220730+0000","last_clean":"2024-04-24T01:23:08.220730+0000","last_became_active":"2024-04-24T00:37:10.785333+0000","last_became_peered":"2024-04-24T00:37:10.785333+0000","last_unstale":"2024-04-24T01:23:08.220730+0000","last_undegraded":"2024-04-24T01:23:08.220730+0000","last_fullsized":"2024-04-24T01:23:08.220730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:24:20.306473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":814,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.221748+0000","last_change":"2024-04-24T00:37:05.801930+0000","last_active":"2024-04-24T01:23:08.221748+0000","last_peered":"2024-04-24T01:23:08.221748+0000","last_clean":"2024-04-24T01:23:08.221748+0000","last_became_active":"2024-04-24T00:37:05.799306+0000","last_became_peered":"2024-04-24T00:37:05.799306+0000","last_unstale":"2024-04-24T01:23:08.221748+0000","last_undegraded":"2024-04-24T01:23:08.221748+0000","last_fullsized":"2024-04-24T01:23:08.221748+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_clean_scrub_stamp":"2024-04-24T00:37:04.744801+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:37:53.503175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738938,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935784,"kb_used_data":2712,"kb_used_omap":148,"kb_used_meta":932907,"kb_avail":93436056,"statfs":{"total":96636764160,"available":95678521344,"internally_reserved":0,"allocated":2777088,"data_stored":1477691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":152420,"internal_metadata":955296924},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738939,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1482920,"kb_used_data":3292,"kb_used_omap":131,"kb_used_meta":1479484,"kb_avail":92888920,"statfs":{"total":96636764160,"available":95118254080,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134892,"internal_metadata":1514991892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738940,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667436,"kb_used_data":3292,"kb_used_omap":220,"kb_used_meta":663907,"kb_avail":93704404,"statfs":{"total":96636764160,"available":95953309696,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225879,"internal_metadata":679841193},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:24:14.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T01:24:14.944 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T01:24:14.944 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":337,"fsid":"155f5f73-fc57-4025-a4b4-fdcde5f1f25e","created":"2024-04-24T00:36:53.725251+0000","modified":"2024-04-24T01:23:08.198421+0000","last_up_change":"2024-04-24T00:37:03.735676+0000","last_in_change":"2024-04-24T00:36:54.643561+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:37:04.722047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:37:09.074949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2368b4cd-28d5-4a67-906e-4559d1b49adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6810","nonce":290552072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6814","nonce":290552072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6821","nonce":290552072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":290552072},{"type":"v1","addr":"172.21.15.67:6817","nonce":290552072}]},"public_addr":"172.21.15.67:6810/290552072","cluster_addr":"172.21.15.67:6814/290552072","heartbeat_back_addr":"172.21.15.67:6821/290552072","heartbeat_front_addr":"172.21.15.67:6817/290552072","state":["exists","up"]},{"osd":1,"uuid":"544e79cb-b3f4-4a0a-b050-10129b035da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6811","nonce":86988388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6816","nonce":86988388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6823","nonce":86988388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":86988388},{"type":"v1","addr":"172.21.15.67:6820","nonce":86988388}]},"public_addr":"172.21.15.67:6811/86988388","cluster_addr":"172.21.15.67:6816/86988388","heartbeat_back_addr":"172.21.15.67:6823/86988388","heartbeat_front_addr":"172.21.15.67:6820/86988388","state":["exists","up"]},{"osd":2,"uuid":"90e497de-f375-4463-be63-b7dadec011e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6801","nonce":1778384542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6803","nonce":1778384542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6807","nonce":1778384542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1778384542},{"type":"v1","addr":"172.21.15.67:6805","nonce":1778384542}]},"public_addr":"172.21.15.67:6801/1778384542","cluster_addr":"172.21.15.67:6803/1778384542","heartbeat_back_addr":"172.21.15.67:6807/1778384542","heartbeat_front_addr":"172.21.15.67:6805/1778384542","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:36:59.908287+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:37:00.010705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:36:59.930058+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/1624989125":"2024-04-24T02:00:18.438046+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T01:24:15.962 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-04-24T01:24:15.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T01:24:16.130 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-24T01:24:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T01:24:16.131 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-24T01:24:16.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-04-24T01:24:16.431 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T01:24:16.448 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-04-24T01:24:16.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T01:24:16.599 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-24T01:24:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T01:24:16.600 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-24T01:24:16.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-04-24T01:24:16.903 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T01:24:16.921 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-04-24T01:24:16.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T01:24:17.059 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-24T01:24:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T01:24:17.059 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-24T01:24:17.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-04-24T01:24:17.353 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T01:24:17.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T01:24:17.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T01:24:17.673 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-24T01:24:17.690 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1751,"stamp":"2024-04-24T01:24:17.668093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3086140,"kb_used_data":9296,"kb_used_omap":501,"kb_used_meta":3076298,"kb_avail":280029380,"statfs":{"total":289910292480,"available":286750085120,"internally_reserved":0,"allocated":9519104,"data_stored":5613809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":513191,"internal_metadata":3150130009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002910"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.220676+0000","last_change":"2024-04-24T00:37:10.788095+0000","last_active":"2024-04-24T01:23:08.220676+0000","last_peered":"2024-04-24T01:23:08.220676+0000","last_clean":"2024-04-24T01:23:08.220676+0000","last_became_active":"2024-04-24T00:37:10.787769+0000","last_became_peered":"2024-04-24T00:37:10.787769+0000","last_unstale":"2024-04-24T01:23:08.220676+0000","last_undegraded":"2024-04-24T01:23:08.220676+0000","last_fullsized":"2024-04-24T01:23:08.220676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:23:01.042116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.223591+0000","last_change":"2024-04-24T00:37:10.788073+0000","last_active":"2024-04-24T01:23:08.223591+0000","last_peered":"2024-04-24T01:23:08.223591+0000","last_clean":"2024-04-24T01:23:08.223591+0000","last_became_active":"2024-04-24T00:37:10.787658+0000","last_became_peered":"2024-04-24T00:37:10.787658+0000","last_unstale":"2024-04-24T01:23:08.223591+0000","last_undegraded":"2024-04-24T01:23:08.223591+0000","last_fullsized":"2024-04-24T01:23:08.223591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:32:27.147352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.222009+0000","last_change":"2024-04-24T00:37:10.785452+0000","last_active":"2024-04-24T01:23:08.222009+0000","last_peered":"2024-04-24T01:23:08.222009+0000","last_clean":"2024-04-24T01:23:08.222009+0000","last_became_active":"2024-04-24T00:37:10.785100+0000","last_became_peered":"2024-04-24T00:37:10.785100+0000","last_unstale":"2024-04-24T01:23:08.222009+0000","last_undegraded":"2024-04-24T01:23:08.222009+0000","last_fullsized":"2024-04-24T01:23:08.222009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:59.597565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.219841+0000","last_change":"2024-04-24T00:37:10.787976+0000","last_active":"2024-04-24T01:23:08.219841+0000","last_peered":"2024-04-24T01:23:08.219841+0000","last_clean":"2024-04-24T01:23:08.219841+0000","last_became_active":"2024-04-24T00:37:10.787561+0000","last_became_peered":"2024-04-24T00:37:10.787561+0000","last_unstale":"2024-04-24T01:23:08.219841+0000","last_undegraded":"2024-04-24T01:23:08.219841+0000","last_fullsized":"2024-04-24T01:23:08.219841+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:30:08.931603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.514147+0000","last_change":"2024-04-24T00:37:10.786163+0000","last_active":"2024-04-24T01:23:08.514147+0000","last_peered":"2024-04-24T01:23:08.514147+0000","last_clean":"2024-04-24T01:23:08.514147+0000","last_became_active":"2024-04-24T00:37:10.785718+0000","last_became_peered":"2024-04-24T00:37:10.785718+0000","last_unstale":"2024-04-24T01:23:08.514147+0000","last_undegraded":"2024-04-24T01:23:08.514147+0000","last_fullsized":"2024-04-24T01:23:08.514147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:57.680592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.517744+0000","last_change":"2024-04-24T00:37:10.786864+0000","last_active":"2024-04-24T01:23:08.517744+0000","last_peered":"2024-04-24T01:23:08.517744+0000","last_clean":"2024-04-24T01:23:08.517744+0000","last_became_active":"2024-04-24T00:37:10.785638+0000","last_became_peered":"2024-04-24T00:37:10.785638+0000","last_unstale":"2024-04-24T01:23:08.517744+0000","last_undegraded":"2024-04-24T01:23:08.517744+0000","last_fullsized":"2024-04-24T01:23:08.517744+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:14.839760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":681,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.513030+0000","last_change":"2024-04-24T00:37:10.786316+0000","last_active":"2024-04-24T01:23:08.513030+0000","last_peered":"2024-04-24T01:23:08.513030+0000","last_clean":"2024-04-24T01:23:08.513030+0000","last_became_active":"2024-04-24T00:37:10.785901+0000","last_became_peered":"2024-04-24T00:37:10.785901+0000","last_unstale":"2024-04-24T01:23:08.513030+0000","last_undegraded":"2024-04-24T01:23:08.513030+0000","last_fullsized":"2024-04-24T01:23:08.513030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:45:47.984157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.220730+0000","last_change":"2024-04-24T00:37:10.785726+0000","last_active":"2024-04-24T01:23:08.220730+0000","last_peered":"2024-04-24T01:23:08.220730+0000","last_clean":"2024-04-24T01:23:08.220730+0000","last_became_active":"2024-04-24T00:37:10.785333+0000","last_became_peered":"2024-04-24T00:37:10.785333+0000","last_unstale":"2024-04-24T01:23:08.220730+0000","last_undegraded":"2024-04-24T01:23:08.220730+0000","last_fullsized":"2024-04-24T01:23:08.220730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:09.767225+0000","last_clean_scrub_stamp":"2024-04-24T00:37:09.767225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:24:20.306473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":814,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:23:08.221748+0000","last_change":"2024-04-24T00:37:05.801930+0000","last_active":"2024-04-24T01:23:08.221748+0000","last_peered":"2024-04-24T01:23:08.221748+0000","last_clean":"2024-04-24T01:23:08.221748+0000","last_became_active":"2024-04-24T00:37:05.799306+0000","last_became_peered":"2024-04-24T00:37:05.799306+0000","last_unstale":"2024-04-24T01:23:08.221748+0000","last_undegraded":"2024-04-24T01:23:08.221748+0000","last_fullsized":"2024-04-24T01:23:08.221748+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:37:04.744801+0000","last_clean_scrub_stamp":"2024-04-24T00:37:04.744801+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:37:53.503175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935784,"kb_used_data":2712,"kb_used_omap":148,"kb_used_meta":932907,"kb_avail":93436056,"statfs":{"total":96636764160,"available":95678521344,"internally_reserved":0,"allocated":2777088,"data_stored":1477691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":152420,"internal_metadata":955296924},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738940,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1482920,"kb_used_data":3292,"kb_used_omap":131,"kb_used_meta":1479484,"kb_avail":92888920,"statfs":{"total":96636764160,"available":95118254080,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134892,"internal_metadata":1514991892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738941,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667436,"kb_used_data":3292,"kb_used_omap":220,"kb_used_meta":663907,"kb_avail":93704404,"statfs":{"total":96636764160,"available":95953309696,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225879,"internal_metadata":679841193},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:24:17.691 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-24T00:37:09.767225+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=9, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=24, tm_sec=14, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:24:17.691 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T00:37:09.767225+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=9, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=24, tm_sec=14, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:24:17.691 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-24T00:37:09.767225+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=9, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=24, tm_sec=14, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:24:17.691 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-24T00:37:09.767225+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=9, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=24, tm_sec=14, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:24:17.691 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-24T00:37:09.767225+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=9, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=24, tm_sec=14, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:24:17.691 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-24T00:37:09.767225+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=9, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=24, tm_sec=14, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:24:17.691 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T00:37:09.767225+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=9, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=24, tm_sec=14, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:24:17.691 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-24T00:37:09.767225+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=9, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=24, tm_sec=14, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:24:17.691 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T00:37:04.744801+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=37, tm_sec=4, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=24, tm_sec=14, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T01:24:17.692 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T01:24:37.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T01:24:37.998 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-24T01:24:37.999 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-24T01:24:38.015 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1761,"stamp":"2024-04-24T01:24:37.673197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3086120,"kb_used_data":9296,"kb_used_omap":501,"kb_used_meta":3076298,"kb_avail":280029400,"statfs":{"total":289910292480,"available":286750105600,"internally_reserved":0,"allocated":9519104,"data_stored":5613809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":513191,"internal_metadata":3150130009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003042"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":688,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:24:22.488566+0000","last_change":"2024-04-24T01:24:22.488589+0000","last_active":"2024-04-24T01:24:22.488566+0000","last_peered":"2024-04-24T01:24:22.488566+0000","last_clean":"2024-04-24T01:24:22.488566+0000","last_became_active":"2024-04-24T00:37:10.787769+0000","last_became_peered":"2024-04-24T00:37:10.787769+0000","last_unstale":"2024-04-24T01:24:22.488566+0000","last_undegraded":"2024-04-24T01:24:22.488566+0000","last_fullsized":"2024-04-24T01:24:22.488566+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:24:22.488468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:24:22.488468+0000","last_clean_scrub_stamp":"2024-04-24T01:24:22.488468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:22:24.941234+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":688,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:24:21.412140+0000","last_change":"2024-04-24T01:24:21.412195+0000","last_active":"2024-04-24T01:24:21.412140+0000","last_peered":"2024-04-24T01:24:21.412140+0000","last_clean":"2024-04-24T01:24:21.412140+0000","last_became_active":"2024-04-24T00:37:10.787658+0000","last_became_peered":"2024-04-24T00:37:10.787658+0000","last_unstale":"2024-04-24T01:24:21.412140+0000","last_undegraded":"2024-04-24T01:24:21.412140+0000","last_fullsized":"2024-04-24T01:24:21.412140+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:24:21.411895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:24:21.411895+0000","last_clean_scrub_stamp":"2024-04-24T01:24:21.411895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:57:58.359076+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":689,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:24:18.604633+0000","last_change":"2024-04-24T01:24:18.604686+0000","last_active":"2024-04-24T01:24:18.604633+0000","last_peered":"2024-04-24T01:24:18.604633+0000","last_clean":"2024-04-24T01:24:18.604633+0000","last_became_active":"2024-04-24T00:37:10.785100+0000","last_became_peered":"2024-04-24T00:37:10.785100+0000","last_unstale":"2024-04-24T01:24:18.604633+0000","last_undegraded":"2024-04-24T01:24:18.604633+0000","last_fullsized":"2024-04-24T01:24:18.604633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:24:18.604384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:24:18.604384+0000","last_clean_scrub_stamp":"2024-04-24T01:24:18.604384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:20.990574+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":689,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:24:20.411280+0000","last_change":"2024-04-24T01:24:20.411334+0000","last_active":"2024-04-24T01:24:20.411280+0000","last_peered":"2024-04-24T01:24:20.411280+0000","last_clean":"2024-04-24T01:24:20.411280+0000","last_became_active":"2024-04-24T00:37:10.787561+0000","last_became_peered":"2024-04-24T00:37:10.787561+0000","last_unstale":"2024-04-24T01:24:20.411280+0000","last_undegraded":"2024-04-24T01:24:20.411280+0000","last_fullsized":"2024-04-24T01:24:20.411280+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:24:20.411068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:24:20.411068+0000","last_clean_scrub_stamp":"2024-04-24T01:24:20.411068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:30:07.006197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":689,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:24:16.624410+0000","last_change":"2024-04-24T01:24:16.624432+0000","last_active":"2024-04-24T01:24:16.624410+0000","last_peered":"2024-04-24T01:24:16.624410+0000","last_clean":"2024-04-24T01:24:16.624410+0000","last_became_active":"2024-04-24T00:37:10.785718+0000","last_became_peered":"2024-04-24T00:37:10.785718+0000","last_unstale":"2024-04-24T01:24:16.624410+0000","last_undegraded":"2024-04-24T01:24:16.624410+0000","last_fullsized":"2024-04-24T01:24:16.624410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:24:16.624310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:24:16.624310+0000","last_clean_scrub_stamp":"2024-04-24T01:24:16.624310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:09.638457+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:24:18.719971+0000","last_change":"2024-04-24T01:24:18.719994+0000","last_active":"2024-04-24T01:24:18.719971+0000","last_peered":"2024-04-24T01:24:18.719971+0000","last_clean":"2024-04-24T01:24:18.719971+0000","last_became_active":"2024-04-24T00:37:10.785638+0000","last_became_peered":"2024-04-24T00:37:10.785638+0000","last_unstale":"2024-04-24T01:24:18.719971+0000","last_undegraded":"2024-04-24T01:24:18.719971+0000","last_fullsized":"2024-04-24T01:24:18.719971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:24:18.719842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:24:18.719842+0000","last_clean_scrub_stamp":"2024-04-24T01:24:18.719842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:14:44.066947+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":690,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:24:17.666506+0000","last_change":"2024-04-24T01:24:17.666629+0000","last_active":"2024-04-24T01:24:17.666506+0000","last_peered":"2024-04-24T01:24:17.666506+0000","last_clean":"2024-04-24T01:24:17.666506+0000","last_became_active":"2024-04-24T00:37:10.785901+0000","last_became_peered":"2024-04-24T00:37:10.785901+0000","last_unstale":"2024-04-24T01:24:17.666506+0000","last_undegraded":"2024-04-24T01:24:17.666506+0000","last_fullsized":"2024-04-24T01:24:17.666506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:24:17.666238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:24:17.666238+0000","last_clean_scrub_stamp":"2024-04-24T01:24:17.666238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:25:45.460178+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":689,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:24:19.611200+0000","last_change":"2024-04-24T01:24:19.611224+0000","last_active":"2024-04-24T01:24:19.611200+0000","last_peered":"2024-04-24T01:24:19.611200+0000","last_clean":"2024-04-24T01:24:19.611200+0000","last_became_active":"2024-04-24T00:37:10.785333+0000","last_became_peered":"2024-04-24T00:37:10.785333+0000","last_unstale":"2024-04-24T01:24:19.611200+0000","last_undegraded":"2024-04-24T01:24:19.611200+0000","last_fullsized":"2024-04-24T01:24:19.611200+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:24:19.611092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:24:19.611092+0000","last_clean_scrub_stamp":"2024-04-24T01:24:19.611092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:10:53.594356+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"337'99","reported_seq":825,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-24T01:24:17.456439+0000","last_change":"2024-04-24T01:24:17.456508+0000","last_active":"2024-04-24T01:24:17.456439+0000","last_peered":"2024-04-24T01:24:17.456439+0000","last_clean":"2024-04-24T01:24:17.456439+0000","last_became_active":"2024-04-24T00:37:05.799306+0000","last_became_peered":"2024-04-24T00:37:05.799306+0000","last_unstale":"2024-04-24T01:24:17.456439+0000","last_undegraded":"2024-04-24T01:24:17.456439+0000","last_fullsized":"2024-04-24T01:24:17.456439+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'99","last_scrub_stamp":"2024-04-24T01:24:17.456233+0000","last_deep_scrub":"337'99","last_deep_scrub_stamp":"2024-04-24T01:24:17.456233+0000","last_clean_scrub_stamp":"2024-04-24T01:24:17.456233+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:11:06.320168+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738943,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935776,"kb_used_data":2712,"kb_used_omap":148,"kb_used_meta":932907,"kb_avail":93436064,"statfs":{"total":96636764160,"available":95678529536,"internally_reserved":0,"allocated":2777088,"data_stored":1477691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":152420,"internal_metadata":955296924},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738944,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1482916,"kb_used_data":3292,"kb_used_omap":131,"kb_used_meta":1479484,"kb_avail":92888924,"statfs":{"total":96636764160,"available":95118258176,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134892,"internal_metadata":1514991892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738945,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667428,"kb_used_data":3292,"kb_used_omap":220,"kb_used_meta":663907,"kb_avail":93704412,"statfs":{"total":96636764160,"available":95953317888,"internally_reserved":0,"allocated":3371008,"data_stored":2068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225879,"internal_metadata":679841193},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:24:38.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-04-24T01:24:38.331 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T01:24:38.332 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T01:24:38.332 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-04-24T01:24:38.332 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:24:43.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2024-04-24T01:24:44.435 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-04-24T01:24:44.435 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-04-24T01:24:44.435 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:24:48.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2024-04-24T01:24:48.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2024-04-24T01:24:50.537 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-04-24T01:24:50.538 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-04-24T01:24:50.538 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:24:53.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-04-24T01:24:53.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2024-04-24T01:24:56.640 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-04-24T01:24:56.640 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T01:24:56.640 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-04-24T01:24:56.640 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:24:56.672 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-04-24T01:24:56.672 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T01:24:56.672 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-04-24T01:24:56.673 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T01:24:56.748 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-04-24T01:24:56.748 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T01:24:56.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-24T01:24:56.871 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T01:24:56.871 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-04-24T01:24:57.343 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T01:24:57.343 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-04-24T01:24:57.704 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T01:24:57.704 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-04-24T01:24:58.039 INFO:tasks.ceph:Archiving mon data... 2024-04-24T01:24:58.039 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535/data/cluster1.mon.a.tgz 2024-04-24T01:24:58.039 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-04-24T01:24:58.056 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.xuhTolIDOV 2024-04-24T01:24:58.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.xuhTolIDOV 2024-04-24T01:24:58.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.xuhTolIDOV 2024-04-24T01:24:58.324 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.xuhTolIDOV is 365KB 2024-04-24T01:24:58.385 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.xuhTolIDOV 2024-04-24T01:24:58.392 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T01:24:58.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T01:24:58.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T01:24:58.516 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T01:24:58.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-04-24T01:24:58.617 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T01:24:58.618 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T01:24:58.620 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535/remote/smithi067/crash 2024-04-24T01:24:58.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T01:24:58.633 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535/remote/smithi134/crash 2024-04-24T01:24:58.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T01:24:58.647 INFO:tasks.ceph:Compressing logs... 2024-04-24T01:24:58.647 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T01:24:58.681 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T01:24:58.701 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28351.log 2024-04-24T01:24:58.701 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30270.log 2024-04-24T01:24:58.719 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28727.log 2024-04-24T01:24:58.719 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28351.log.gz 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30270.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30270.log.gz 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30296.log 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28727.log.gz 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28653.log 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.30296.log: -- /var/log/ceph/cluster1-client.admin.28723.log 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30296.log.gz 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23974.log 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28653.log.gz 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24092.log 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28723.log.gz 2024-04-24T01:24:58.720 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.23974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28463.log 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.23974.log.gz 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.24092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30556.log 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24092.log.gz 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28575.log 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28463.log.gz 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30556.log.gz 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24068.log 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28575.log.gz 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30374.log 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.24068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30530.log 2024-04-24T01:24:58.721 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24068.log.gz 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28652.log 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30374.log.gz 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24116.log 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30530.log.gz 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30478.log 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28652.log.gz 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.24116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24116.log.gz 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28489.log 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30478.log.gz 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-04-24T01:24:58.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28489.log.gz 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28541.log 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30580.log 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30504.log 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28541.log.gz 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30400.log 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30580.log.gz 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28801.log 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30504.log.gz 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30426.log 2024-04-24T01:24:58.723 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30400.log.gz 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24219.log 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28801.log.gz 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28853.log 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30426.log.gz 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.24219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30452.log 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24219.log.gz 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30322.log 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28853.log.gz 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28571.log 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30452.log.gz 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28827.log 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30322.log.gz 2024-04-24T01:24:58.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24151.log 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28571.log.gz 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28412.log 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28827.log.gz 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.24151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28573.log 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24151.log.gz 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28732.log 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28412.log.gz 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28437.log 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28573.log.gz 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28732.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20343.log 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28732.log.gz 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28437.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-04-24T01:24:58.725 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28437.log.gz 2024-04-24T01:24:58.726 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.tmp-client.admin.20343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30348.log 2024-04-24T01:24:58.726 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20343.log.gz 2024-04-24T01:24:58.726 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28645.log 2024-04-24T01:24:58.726 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.30348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28515.log 2024-04-24T01:24:58.726 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30348.log.gz 2024-04-24T01:24:58.726 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-04-24T01:24:58.726 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28645.log.gz 2024-04-24T01:24:58.726 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28515.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28515.log.gz 2024-04-24T01:24:58.726 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-osd.1.log: 88.0% -- replaced with /var/log/ceph/cluster1.log.gz 2024-04-24T01:24:58.736 INFO:teuthology.orchestra.run.smithi067.stderr: 93.0% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-04-24T01:24:58.762 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141804.log 2024-04-24T01:24:58.762 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131419.log 2024-04-24T01:24:58.763 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138845.log 2024-04-24T01:24:58.763 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141804.log.gz 2024-04-24T01:24:58.763 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74510.log 2024-04-24T01:24:58.763 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48190.log 2024-04-24T01:24:58.763 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138845.log: /var/log/ceph/cluster2-client.mirror.74510.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38959.log 2024-04-24T01:24:58.763 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.131419.log.gz 2024-04-24T01:24:58.763 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56146.log 2024-04-24T01:24:58.763 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38959.log: gzip -5 --verbose -- 87.6% /var/log/ceph/cluster2-client.mirror.42221.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.138845.log.gz 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137107.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56375.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65601.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137107.log: /var/log/ceph/cluster2-client.mirror.56375.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85764.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137107.log.gz 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84343.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97946.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.84343.log: --verbose -- /var/log/ceph/cluster2-client.mirror.61549.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132336.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138326.log 2024-04-24T01:24:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68006.log 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132336.log: /var/log/ceph/cluster2-client.mirror.138326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88861.log 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141196.log 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125729.log 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88861.log: /var/log/ceph/cluster2-client.mirror.141196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115257.log 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.138326.log.gz 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.141196.log.gz 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46832.log 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118912.log 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125729.log.gz 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132336.log.gz 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46832.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115257.log.gz 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123923.log 2024-04-24T01:24:58.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40476.log 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118912.log.gz 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78570.log 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46832.log.gz 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123923.log.gz 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66991.log 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110402.log 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78360.log 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62554.log 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87286.log 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83461.log 2024-04-24T01:24:58.766 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34894.log 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.110402.log.gz 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74510.log.gz 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131891.log 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83461.log: /var/log/ceph/cluster2-client.mirror.34894.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75544.log 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74720.log 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131720.log 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131891.log.gz 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103186.log 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85589.log 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67201.log 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103186.log.gz 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95218.log 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74720.log.gz 2024-04-24T01:24:58.767 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/cluster2-client.mirror.67201.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136814.log 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47802.log 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.131720.log.gz 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139994.log 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136814.log.gz 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47802.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38959.log.gz 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114352.log 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87916.log 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.114352.log: -- replaced with /var/log/ceph/cluster1-client.mirror.139994.log.gz 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107052.log 2024-04-24T01:24:58.768 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114352.log.gz 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68846.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81934.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31949.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81934.log: 85.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48190.log.gz 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68951.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51001.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134765.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50745.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133903.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119362.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134765.log.gz 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30384.log 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133903.log.gz 2024-04-24T01:24:58.769 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140100.log 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119362.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84343.log.gz 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.30384.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119362.log.gz 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103905.log 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53950.log 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140100.log: /var/log/ceph/cluster2-client.mirror.103905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95929.log 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140100.log.gz 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53950.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.103905.log.gz 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138624.log 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42663.log 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95929.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62554.log.gz 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61024.log 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136117.log 2024-04-24T01:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133994.log 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61024.log: /var/log/ceph/cluster1-client.mirror.136117.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94038.log 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136117.log.gz 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.138624.log.gz 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90233.log 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135824.log 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94038.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31949.log.gz 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124052.log 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112289.log 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135824.log: /var/log/ceph/cluster2-client.mirror.124052.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135824.log.gz 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135218.log 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133994.log.gz 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124052.log.gz 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132759.log 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68846.log.gz 2024-04-24T01:24:58.771 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.112289.log.gz 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33123.log 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.135218.log.gz 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68951.log.gz 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105585.log 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.33123.log: /var/log/ceph/cluster2-client.mirror.92053.log 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129753.log 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105585.log.gz 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49546.log 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132759.log.gz 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116636.log 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129753.log.gz 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137197.log 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32321.log 2024-04-24T01:24:58.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137197.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116636.log.gz 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108223.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137197.log.gz 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42413.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107641.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108223.log.gz 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121613.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65741.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107641.log: /var/log/ceph/cluster2-client.mirror.121613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29724.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107641.log.gz 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65741.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.121613.log.gz 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92158.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.29724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136927.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133810.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31821.log 2024-04-24T01:24:58.773 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136927.log.gz 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133810.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67201.log.gz 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.102278.log 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31821.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49643.log 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133810.log.gz 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33158.log 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.2.102278.log: /var/log/ceph/cluster2-client.mirror.49643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66886.log 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48867.log 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66886.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77835.log 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98643.log 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33123.log.gz 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61788.log 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79169.log 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98643.log.gz 2024-04-24T01:24:58.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135092.log 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50516.log 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79169.log: /var/log/ceph/cluster2-client.mirror.135092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108177.log 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124446.log 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127851.log 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108177.log: /var/log/ceph/cluster2-client.mirror.124446.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108177.log.gz 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124446.log.gz 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.135092.log.gz 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81409.log 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118358.log 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127851.log.gz 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81409.log: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30384.log.gz 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118358.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118358.log.gz 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89705.log 2024-04-24T01:24:58.775 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117812.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90968.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.117812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73771.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.117812.log.gz 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46256.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51133.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46256.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59203.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94038.log.gz 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51133.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107411.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67096.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34743.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67096.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107411.log.gz 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63491.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112343.log 2024-04-24T01:24:58.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34743.log: /var/log/ceph/cluster2-client.mirror.63491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107446.log 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31449.log 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112343.log.gz 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107446.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106583.log 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53950.log.gz 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31449.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107446.log.gz 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81584.log 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46291.log 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81584.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106583.log.gz 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105284.log 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.46291.log: -- /var/log/ceph/cluster1-client.mirror.136904.log 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105284.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56146.log.gz 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107870.log 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105284.log.gz 2024-04-24T01:24:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136904.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56375.log.gz 2024-04-24T01:24:58.778 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109264.log 2024-04-24T01:24:58.778 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136904.log.gz 2024-04-24T01:24:58.778 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45903.log 2024-04-24T01:24:58.778 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107870.log: /var/log/ceph/cluster2-client.mirror.109264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130141.log 2024-04-24T01:24:58.778 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109264.log.gz 2024-04-24T01:24:58.778 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45903.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107870.log.gz 2024-04-24T01:24:58.778 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45127.log 2024-04-24T01:24:58.796 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141421.log 2024-04-24T01:24:58.796 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135284.log 2024-04-24T01:24:58.796 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/cluster2-client.mirror.141421.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137354.log 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135284.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141421.log.gz 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.135284.log.gz 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59028.log 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77905.log 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9%/var/log/ceph/cluster2-client.mirror.59028.log: -- replaced with /var/log/ceph/cluster1-client.mirror.137354.log.gz 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53139.log 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.29724.log 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53139.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130141.log.gz 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83760.log 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.3.29724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131913.log 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101340.log 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121648.log 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131913.log.gz 2024-04-24T01:24:58.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29170.log 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/cluster2-client.mirror.121648.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140929.log 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29170.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121648.log.gz 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29170.log.gz 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35766.log 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45189.log 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140929.log.gz 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42329.log 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45189.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85764.log.gz 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77310.log 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142012.log 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86709.log 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.142012.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142012.log.gz 2024-04-24T01:24:58.798 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53562.log 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.86709.log: --verbose -- /var/log/ceph/cluster2-client.mirror.62589.log 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53562.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100048.log 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108396.log 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57094.log 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136679.log 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57094.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108396.log.gz 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.90233.log.gz 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128625.log 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136679.log: gzip 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136679.log.gz 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59448.log 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128625.log: gzip -5 --verbose -- 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128625.log.gz 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr: /var/log/ceph/cluster1-client.mirror.103476.log 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59448.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47802.log.gz 2024-04-24T01:24:58.799 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.103476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140430.log 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51424.log 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140430.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.140430.log.gz 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103476.log.gz 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95761.log 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51424.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34894.log.gz 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123761.log 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136207.log 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122082.log 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.123761.log.gz 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136207.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136207.log.gz 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133043.log 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122082.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132174.log 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40476.log.gz 2024-04-24T01:24:58.800 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.122082.log.gz 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133043.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133043.log.gz 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125498.log 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95887.log 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132174.log: /var/log/ceph/cluster1-client.mirror.125498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51195.log 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95887.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.132174.log.gz 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95929.log.gz 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125498.log.gz 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33158.log.gz 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66991.log.gz 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93828.log 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104404.log 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106695.log 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85379.log 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.106695.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104404.log.gz 2024-04-24T01:24:58.801 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42663.log.gz 2024-04-24T01:24:58.806 INFO:teuthology.orchestra.run.smithi134.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.102278.log.gz 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95304.log 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85379.log: 91.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117283.log 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.106695.log.gz 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138186.log 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.117283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91808.log 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112254.log 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.117283.log.gz 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91808.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92053.log.gz 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119006.log 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92158.log.gz 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132123.log 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34243.log 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.112254.log.gz 2024-04-24T01:24:58.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119006.log: /var/log/ceph/cluster2-client.mirror.132123.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95761.log.gz 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119006.log.gz 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.138186.log.gz 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70985.log 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34243.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105070.log 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48673.log 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95887.log.gz 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93828.log.gz 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105070.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105070.log.gz 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132123.log.gz 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110873.log 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96736.log 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.110873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75988.log 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33030.log 2024-04-24T01:24:58.808 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110873.log.gz 2024-04-24T01:24:58.809 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.75988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28984.log 2024-04-24T01:24:58.809 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39118.log 2024-04-24T01:24:58.809 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.28984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123045.log 2024-04-24T01:24:58.809 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28984.log.gz 2024-04-24T01:24:58.809 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49546.log.gz 2024-04-24T01:24:58.809 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49643.log.gz 2024-04-24T01:24:58.809 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140607.log 2024-04-24T01:24:58.809 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39118.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123045.log.gz 2024-04-24T01:24:58.809 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124889.log 2024-04-24T01:24:58.815 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63631.log 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr: 84.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140607.log.gz 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29166.log 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124889.log.gz 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58105.log 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129854.log 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29166.log: /var/log/ceph/cluster2-client.mirror.58105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69616.log 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29166.log.gz 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29088.log 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75544.log.gz 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69616.log: gzip -5 --verbose -- 88.2% /var/log/ceph/cluster2-client.admin.29549.log 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.129854.log.gz 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35359.log 2024-04-24T01:24:58.816 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29088.log.gz 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81409.log.gz 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139640.log 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.81584.log.gz 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29549.log.gz 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128803.log 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35359.log: /var/log/ceph/cluster2-client.mirror.139640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139450.log 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76575.log 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.139640.log.gz 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128803.log.gz 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82844.log 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139450.log: /var/log/ceph/cluster2-client.mirror.76575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123589.log 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116284.log 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.139450.log.gz 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.123589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118630.log 2024-04-24T01:24:58.817 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89390.log 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118630.log: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40185.log 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116284.log.gz 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118630.log.gz 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39409.log 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123589.log.gz 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116697.log 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40185.log: /var/log/ceph/cluster2-client.mirror.39409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30123.log 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115292.log 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.30123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130808.log 2024-04-24T01:24:58.818 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50516.log.gz 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115292.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50745.log.gz 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115292.log.gz 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51001.log.gz 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116697.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.51133.log.gz 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51195.log.gz 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37601.log 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66886.log.gz 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106529.log 2024-04-24T01:24:58.820 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52848.log 2024-04-24T01:24:58.821 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35766.log.gz 2024-04-24T01:24:58.821 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106529.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130808.log.gz 2024-04-24T01:24:58.821 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106529.log.gz 2024-04-24T01:24:58.822 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48999.log 2024-04-24T01:24:58.823 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69721.log 2024-04-24T01:24:58.823 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24253.log 2024-04-24T01:24:58.823 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125001.log 2024-04-24T01:24:58.825 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.24253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49061.log 2024-04-24T01:24:58.825 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24253.log.gz 2024-04-24T01:24:58.825 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.125001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127191.log 2024-04-24T01:24:58.825 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107030.log 2024-04-24T01:24:58.825 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127191.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126128.log 2024-04-24T01:24:58.826 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59868.log 2024-04-24T01:24:58.826 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.127191.log.gz 2024-04-24T01:24:58.827 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.107030.log: /var/log/ceph/cluster2-client.mirror.126128.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120388.log 2024-04-24T01:24:58.827 INFO:teuthology.orchestra.run.smithi134.stderr: 92.1%/var/log/ceph/cluster2-client.mirror.59868.log: -- replaced with /var/log/ceph/cluster1-client.mirror.125001.log.gz 2024-04-24T01:24:58.827 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127089.log 2024-04-24T01:24:58.827 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103049.log 2024-04-24T01:24:58.828 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127089.log: 91.8% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.127089.log.gz 2024-04-24T01:24:58.828 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.126128.log.gz 2024-04-24T01:24:58.828 INFO:teuthology.orchestra.run.smithi134.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107030.log.gz 2024-04-24T01:24:58.828 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141326.log 2024-04-24T01:24:58.829 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72091.log 2024-04-24T01:24:58.829 INFO:teuthology.orchestra.run.smithi134.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.120388.log.gz 2024-04-24T01:24:58.829 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141326.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103049.log.gz 2024-04-24T01:24:58.830 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104555.log 2024-04-24T01:24:58.830 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138509.log 2024-04-24T01:24:58.830 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104555.log: 95.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51327.log 2024-04-24T01:24:58.830 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.97946.log.gz 2024-04-24T01:24:58.830 INFO:teuthology.orchestra.run.smithi134.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141326.log.gz 2024-04-24T01:24:58.831 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.138509.log: -- replaced with /var/log/ceph/cluster2-client.mirror.104555.log.gz 2024-04-24T01:24:58.831 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93618.log 2024-04-24T01:24:58.831 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58070.log 2024-04-24T01:24:58.832 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65636.log 2024-04-24T01:24:58.832 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93618.log: /var/log/ceph/cluster2-client.mirror.58070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113596.log 2024-04-24T01:24:58.833 INFO:teuthology.orchestra.run.smithi134.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.138509.log.gz 2024-04-24T01:24:58.833 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94956.log 2024-04-24T01:24:58.833 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65636.log: /var/log/ceph/cluster2-client.mirror.113596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29474.log 2024-04-24T01:24:58.833 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113596.log.gz 2024-04-24T01:24:58.833 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.94956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34336.log 2024-04-24T01:24:58.833 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57136.log 2024-04-24T01:24:58.833 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.29474.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29474.log.gz 2024-04-24T01:24:58.833 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31856.log 2024-04-24T01:24:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138285.log 2024-04-24T01:24:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57136.log: /var/log/ceph/cluster2-client.mirror.31856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138645.log 2024-04-24T01:24:58.835 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61024.log.gz 2024-04-24T01:24:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69476.log 2024-04-24T01:24:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138347.log 2024-04-24T01:24:58.835 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.138285.log.gz 2024-04-24T01:24:58.836 INFO:teuthology.orchestra.run.smithi134.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.138645.log.gz 2024-04-24T01:24:58.836 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/cluster2-client.mirror.69476.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97992.log 2024-04-24T01:24:58.836 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109164.log 2024-04-24T01:24:58.836 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138347.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.138347.log.gz 2024-04-24T01:24:58.836 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80429.log 2024-04-24T01:24:58.836 INFO:teuthology.orchestra.run.smithi134.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97992.log.gz 2024-04-24T01:24:58.836 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97087.log 2024-04-24T01:24:58.838 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55370.log 2024-04-24T01:24:58.838 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80429.log: /var/log/ceph/cluster2-client.mirror.97087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37795.log 2024-04-24T01:24:58.838 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137467.log 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55370.log: /var/log/ceph/cluster2-client.mirror.37795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40705.log 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129939.log 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137467.log: /var/log/ceph/cluster2-client.mirror.40705.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137467.log.gz 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128520.log 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138241.log 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129939.log: /var/log/ceph/cluster2-client.mirror.128520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88021.log 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129939.log.gz 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69616.log.gz 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128520.log.gz 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58458.log 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138241.log.gz 2024-04-24T01:24:58.839 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37760.log/var/log/ceph/cluster2-client.mirror.88021.log: 2024-04-24T01:24:58.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40123.log 2024-04-24T01:24:58.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120829.log 2024-04-24T01:24:58.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52654.log 2024-04-24T01:24:58.841 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112666.log 2024-04-24T01:24:58.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120829.log: /var/log/ceph/cluster2-client.mirror.52654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141016.log 2024-04-24T01:24:58.842 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120829.log.gz 2024-04-24T01:24:58.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121828.log 2024-04-24T01:24:58.842 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112666.log.gz 2024-04-24T01:24:58.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.141016.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141016.log.gz 2024-04-24T01:24:58.843 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132458.log 2024-04-24T01:24:58.843 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121828.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116319.log 2024-04-24T01:24:58.843 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132458.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132458.log.gz 2024-04-24T01:24:58.844 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32321.log.gz 2024-04-24T01:24:58.846 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91843.log 2024-04-24T01:24:58.847 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60779.log 2024-04-24T01:24:58.847 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116319.log: /var/log/ceph/cluster2-client.mirror.91843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55952.log 2024-04-24T01:24:58.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49969.log 2024-04-24T01:24:58.849 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116319.log.gz 2024-04-24T01:24:58.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66746.log 2024-04-24T01:24:58.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49969.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83159.log 2024-04-24T01:24:58.849 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140700.log 2024-04-24T01:24:58.850 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66746.log: /var/log/ceph/cluster2-client.mirror.83159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86079.log 2024-04-24T01:24:58.850 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88651.log 2024-04-24T01:24:58.850 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140700.log: gzip/var/log/ceph/cluster2-client.mirror.86079.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120546.log 2024-04-24T01:24:58.850 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88651.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.140700.log.gz 2024-04-24T01:24:58.851 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52848.log.gz 2024-04-24T01:24:58.851 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105319.log 2024-04-24T01:24:58.852 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136612.log 2024-04-24T01:24:58.852 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120546.log: /var/log/ceph/cluster2-client.mirror.105319.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67096.log.gz 2024-04-24T01:24:58.852 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120546.log.gz 2024-04-24T01:24:58.852 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105319.log.gz 2024-04-24T01:24:58.852 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39118.log.gz 2024-04-24T01:24:58.854 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48673.log.gz 2024-04-24T01:24:58.854 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48867.log.gz 2024-04-24T01:24:58.855 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140202.log 2024-04-24T01:24:58.855 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78640.log 2024-04-24T01:24:58.857 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136612.log.gz 2024-04-24T01:24:58.857 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115334.log 2024-04-24T01:24:58.857 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140451.log 2024-04-24T01:24:58.857 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78640.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.140202.log.gz 2024-04-24T01:24:58.857 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115334.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89600.log 2024-04-24T01:24:58.857 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115334.log.gz 2024-04-24T01:24:58.857 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62344.log 2024-04-24T01:24:58.859 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140451.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71090.log 2024-04-24T01:24:58.859 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140451.log.gz 2024-04-24T01:24:58.860 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89600.log: /var/log/ceph/cluster2-client.mirror.62344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118877.log 2024-04-24T01:24:58.860 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47837.log 2024-04-24T01:24:58.860 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65111.log 2024-04-24T01:24:58.860 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118877.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108800.log 2024-04-24T01:24:58.860 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:58.860 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47837.log: /var/log/ceph/cluster2-client.mirror.65111.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63491.log.gz 2024-04-24T01:24:58.860 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63631.log.gz 2024-04-24T01:24:58.860 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.139499.log 2024-04-24T01:24:58.861 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29449.log 2024-04-24T01:24:58.861 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108800.log.gz 2024-04-24T01:24:58.861 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.139499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49255.log 2024-04-24T01:24:58.861 INFO:teuthology.orchestra.run.smithi134.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.30123.log.gz 2024-04-24T01:24:58.861 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.29449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29449.log.gz 2024-04-24T01:24:58.862 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133881.log 2024-04-24T01:24:58.862 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99962.log 2024-04-24T01:24:58.862 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133881.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133881.log.gz 2024-04-24T01:24:58.863 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48999.log.gz 2024-04-24T01:24:58.863 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60394.log 2024-04-24T01:24:58.863 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99962.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51424.log.gz 2024-04-24T01:24:58.864 INFO:teuthology.orchestra.run.smithi134.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.139499.log.gz 2024-04-24T01:24:58.864 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45903.log.gz 2024-04-24T01:24:58.865 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46256.log.gz 2024-04-24T01:24:58.866 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46291.log.gz 2024-04-24T01:24:58.866 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.91808.log.gz 2024-04-24T01:24:58.866 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91843.log.gz 2024-04-24T01:24:58.867 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107052.log.gz 2024-04-24T01:24:58.867 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84763.log 2024-04-24T01:24:58.868 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31728.log 2024-04-24T01:24:58.868 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.89705.log.gz 2024-04-24T01:24:58.868 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77590.log 2024-04-24T01:24:58.868 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30106.log 2024-04-24T01:24:58.869 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32042.log 2024-04-24T01:24:58.869 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.30106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69931.log 2024-04-24T01:24:58.869 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122515.log 2024-04-24T01:24:58.870 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39735.log 2024-04-24T01:24:58.870 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115545.log 2024-04-24T01:24:58.870 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39735.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122515.log.gz 2024-04-24T01:24:58.871 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112844.log 2024-04-24T01:24:58.871 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57620.log 2024-04-24T01:24:58.872 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112844.log: 88.2% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45127.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.45189.log.gz 2024-04-24T01:24:58.872 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:58.872 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112844.log.gz 2024-04-24T01:24:58.875 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67376.log 2024-04-24T01:24:58.875 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108698.log 2024-04-24T01:24:58.875 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85167.log 2024-04-24T01:24:58.876 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67376.log: /var/log/ceph/cluster2-client.mirror.108698.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108698.log.gz 2024-04-24T01:24:58.876 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66357.log 2024-04-24T01:24:58.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134483.log 2024-04-24T01:24:58.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52786.log 2024-04-24T01:24:58.878 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134483.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134483.log.gz 2024-04-24T01:24:58.879 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127124.log 2024-04-24T01:24:58.879 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90338.log 2024-04-24T01:24:58.879 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127124.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.127124.log.gz 2024-04-24T01:24:58.880 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56631.log 2024-04-24T01:24:58.880 INFO:teuthology.orchestra.run.smithi134.stderr: 97.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115545.log.gz 2024-04-24T01:24:58.880 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90338.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49255.log.gz 2024-04-24T01:24:58.882 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99109.log 2024-04-24T01:24:58.883 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103990.log 2024-04-24T01:24:58.883 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46929.log 2024-04-24T01:24:58.883 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103990.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103990.log.gz 2024-04-24T01:24:58.884 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54823.log 2024-04-24T01:24:58.885 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94711.log 2024-04-24T01:24:58.885 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109660.log 2024-04-24T01:24:58.886 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74650.log 2024-04-24T01:24:58.886 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109660.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109660.log.gz 2024-04-24T01:24:58.887 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120581.log 2024-04-24T01:24:58.887 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74650.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79589.log 2024-04-24T01:24:58.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141301.log 2024-04-24T01:24:58.888 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120581.log.gz 2024-04-24T01:24:58.889 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140867.log 2024-04-24T01:24:58.889 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.141301.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.141301.log.gz 2024-04-24T01:24:58.889 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95456.log 2024-04-24T01:24:58.890 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128949.log 2024-04-24T01:24:58.892 INFO:teuthology.orchestra.run.smithi134.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140867.log.gz 2024-04-24T01:24:58.893 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134736.log 2024-04-24T01:24:58.894 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128949.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128949.log.gz 2024-04-24T01:24:58.895 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94138.log 2024-04-24T01:24:58.897 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134736.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134736.log.gz 2024-04-24T01:24:58.897 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115868.log 2024-04-24T01:24:58.898 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92953.log 2024-04-24T01:24:58.898 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94138.log: /var/log/ceph/cluster2-client.mirror.115868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76785.log 2024-04-24T01:24:58.899 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55987.log 2024-04-24T01:24:58.900 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93688.log 2024-04-24T01:24:58.900 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62665.log 2024-04-24T01:24:58.901 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77380.log 2024-04-24T01:24:58.902 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63701.log 2024-04-24T01:24:58.903 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96097.log 2024-04-24T01:24:58.904 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65601.log.gz 2024-04-24T01:24:58.904 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65741.log.gz 2024-04-24T01:24:58.905 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63701.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70985.log.gz 2024-04-24T01:24:58.905 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71090.log.gz 2024-04-24T01:24:58.907 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77835.log.gz 2024-04-24T01:24:58.907 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77905.log.gz 2024-04-24T01:24:58.907 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135712.log 2024-04-24T01:24:58.907 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78360.log.gz 2024-04-24T01:24:58.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96097.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85379.log.gz 2024-04-24T01:24:58.909 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85589.log.gz 2024-04-24T01:24:58.909 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101223.log 2024-04-24T01:24:58.910 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135712.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.135712.log.gz 2024-04-24T01:24:58.910 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.90968.log.gz 2024-04-24T01:24:58.910 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110599.log 2024-04-24T01:24:58.911 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101223.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31856.log.gz 2024-04-24T01:24:58.911 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31821.log.gz 2024-04-24T01:24:58.911 INFO:teuthology.orchestra.run.smithi134.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101223.log.gz 2024-04-24T01:24:58.912 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78255.log 2024-04-24T01:24:58.912 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110599.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110599.log.gz 2024-04-24T01:24:58.912 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51327.log.gz 2024-04-24T01:24:58.913 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66746.log.gz 2024-04-24T01:24:58.913 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37601.log.gz 2024-04-24T01:24:58.913 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58105.log.gz 2024-04-24T01:24:58.914 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62344.log.gz 2024-04-24T01:24:58.914 INFO:teuthology.orchestra.run.smithi134.stderr: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.30106.log.gz 2024-04-24T01:24:58.917 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59203.log.gz 2024-04-24T01:24:58.918 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59028.log.gz 2024-04-24T01:24:58.918 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59448.log.gz 2024-04-24T01:24:58.918 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61549.log.gz 2024-04-24T01:24:58.919 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61788.log.gz 2024-04-24T01:24:58.920 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68006.log.gz 2024-04-24T01:24:58.920 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60989.log 2024-04-24T01:24:58.921 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.98243.log 2024-04-24T01:24:58.922 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60989.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37795.log.gz 2024-04-24T01:24:58.922 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135892.log 2024-04-24T01:24:58.923 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.98243.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87286.log.gz 2024-04-24T01:24:58.925 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.88021.log.gz 2024-04-24T01:24:58.925 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87916.log.gz 2024-04-24T01:24:58.926 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88651.log.gz 2024-04-24T01:24:58.927 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88861.log.gz 2024-04-24T01:24:58.929 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89600.log.gz 2024-04-24T01:24:58.929 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65636.log.gz 2024-04-24T01:24:58.929 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78255.log.gz 2024-04-24T01:24:58.929 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.89390.log.gz 2024-04-24T01:24:58.931 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73771.log.gz 2024-04-24T01:24:58.932 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74650.log.gz 2024-04-24T01:24:58.934 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78570.log.gz 2024-04-24T01:24:58.934 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78640.log.gz 2024-04-24T01:24:58.935 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79169.log.gz 2024-04-24T01:24:58.935 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79589.log.gz 2024-04-24T01:24:58.937 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81934.log.gz 2024-04-24T01:24:58.938 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58070.log.gz 2024-04-24T01:24:58.939 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58458.log.gz 2024-04-24T01:24:58.939 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83461.log.gz 2024-04-24T01:24:58.940 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63701.log.gz 2024-04-24T01:24:58.941 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65111.log.gz 2024-04-24T01:24:58.942 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106769.log 2024-04-24T01:24:58.943 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135892.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135892.log.gz 2024-04-24T01:24:58.943 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76575.log.gz 2024-04-24T01:24:58.944 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77310.log.gz 2024-04-24T01:24:58.944 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77380.log.gz 2024-04-24T01:24:58.944 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77590.log.gz 2024-04-24T01:24:58.945 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52654.log.gz 2024-04-24T01:24:58.945 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52786.log.gz 2024-04-24T01:24:58.946 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56631.log.gz 2024-04-24T01:24:58.946 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49061.log.gz 2024-04-24T01:24:58.951 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.admin.75988.log.gz 2024-04-24T01:24:58.959 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118161.log 2024-04-24T01:24:58.959 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106769.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106769.log.gz 2024-04-24T01:24:58.961 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83159.log.gz 2024-04-24T01:24:58.968 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39409.log.gz 2024-04-24T01:24:58.969 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92368.log 2024-04-24T01:24:58.970 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118161.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37760.log.gz 2024-04-24T01:24:58.970 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118161.log.gz 2024-04-24T01:24:58.972 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54823.log.gz 2024-04-24T01:24:58.976 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34743.log.gz 2024-04-24T01:24:58.976 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.95218.log.gz 2024-04-24T01:24:58.977 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35359.log.gz 2024-04-24T01:24:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102944.log 2024-04-24T01:24:58.978 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33030.log.gz 2024-04-24T01:24:58.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92368.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86709.log.gz 2024-04-24T01:24:58.982 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55370.log.gz 2024-04-24T01:24:58.983 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55952.log.gz 2024-04-24T01:24:58.984 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46929.log.gz 2024-04-24T01:24:58.985 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53139.log.gz 2024-04-24T01:24:58.986 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.90338.log.gz 2024-04-24T01:24:58.986 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93618.log.gz 2024-04-24T01:24:58.987 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47837.log.gz 2024-04-24T01:24:58.990 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84763.log.gz 2024-04-24T01:24:58.990 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59833.log 2024-04-24T01:24:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.102944.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.102944.log.gz 2024-04-24T01:24:58.991 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49969.log.gz 2024-04-24T01:24:58.992 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31449.log.gz 2024-04-24T01:24:58.992 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31728.log.gz 2024-04-24T01:24:58.993 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32042.log.gz 2024-04-24T01:24:58.994 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57620.log.gz 2024-04-24T01:24:58.995 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62665.log.gz 2024-04-24T01:24:58.996 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62589.log.gz 2024-04-24T01:24:58.996 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80429.log.gz 2024-04-24T01:24:59.000 INFO:teuthology.orchestra.run.smithi134.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99109.log.gz 2024-04-24T01:24:59.000 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112932.log 2024-04-24T01:24:59.003 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62700.log 2024-04-24T01:24:59.004 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112932.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112932.log.gz 2024-04-24T01:24:59.004 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97192.log 2024-04-24T01:24:59.005 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29799.log 2024-04-24T01:24:59.005 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39541.log 2024-04-24T01:24:59.006 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.29799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47546.log 2024-04-24T01:24:59.009 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39541.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94711.log.gz 2024-04-24T01:24:59.010 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96097.log.gz 2024-04-24T01:24:59.011 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102763.log 2024-04-24T01:24:59.011 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70740.log 2024-04-24T01:24:59.012 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.102763.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102763.log.gz 2024-04-24T01:24:59.012 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97380.log 2024-04-24T01:24:59.013 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70740.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121828.log.gz 2024-04-24T01:24:59.013 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90058.log 2024-04-24T01:24:59.014 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38633.log 2024-04-24T01:24:59.015 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139702.log 2024-04-24T01:24:59.015 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36322.log 2024-04-24T01:24:59.015 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84553.log 2024-04-24T01:24:59.016 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36322.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139702.log.gz 2024-04-24T01:24:59.016 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30367.log 2024-04-24T01:24:59.016 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72091.log.gz 2024-04-24T01:24:59.017 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86604.log 2024-04-24T01:24:59.017 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.30367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132972.log 2024-04-24T01:24:59.017 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86604.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104832.log 2024-04-24T01:24:59.018 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122652.log 2024-04-24T01:24:59.018 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104832.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104832.log.gz 2024-04-24T01:24:59.018 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105730.log 2024-04-24T01:24:59.019 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122652.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132972.log.gz 2024-04-24T01:24:59.019 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122652.log.gz 2024-04-24T01:24:59.019 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90478.log 2024-04-24T01:24:59.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.105730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80728.log 2024-04-24T01:24:59.021 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90478.log: 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105730.log.gz 2024-04-24T01:24:59.022 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91108.log 2024-04-24T01:24:59.022 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79519.log 2024-04-24T01:24:59.022 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85554.log 2024-04-24T01:24:59.023 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91108.log: /var/log/ceph/cluster2-client.mirror.79519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40088.log 2024-04-24T01:24:59.023 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131327.log 2024-04-24T01:24:59.024 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86814.log 2024-04-24T01:24:59.024 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83630.log 2024-04-24T01:24:59.025 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86814.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131327.log.gz 2024-04-24T01:24:59.025 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111461.log 2024-04-24T01:24:59.025 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83630.log: /var/log/ceph/cluster1-client.mirror.111461.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111461.log.gz 2024-04-24T01:24:59.026 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72721.log 2024-04-24T01:24:59.027 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90023.log 2024-04-24T01:24:59.027 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38571.log 2024-04-24T01:24:59.027 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117328.log 2024-04-24T01:24:59.028 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90023.log: /var/log/ceph/cluster2-client.mirror.38571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135442.log 2024-04-24T01:24:59.028 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93688.log.gz 2024-04-24T01:24:59.030 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.117328.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.117328.log.gz 2024-04-24T01:24:59.030 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113756.log 2024-04-24T01:24:59.031 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132052.log 2024-04-24T01:24:59.031 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135442.log.gz 2024-04-24T01:24:59.032 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41481.log 2024-04-24T01:24:59.032 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132052.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113756.log.gz 2024-04-24T01:24:59.032 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72511.log 2024-04-24T01:24:59.033 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85799.log 2024-04-24T01:24:59.033 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72511.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55987.log.gz 2024-04-24T01:24:59.034 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132052.log.gz 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.90478.log.gz 2024-04-24T01:24:59.034 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:59.035 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44122.log 2024-04-24T01:24:59.036 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85799.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70740.log.gz 2024-04-24T01:24:59.036 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40317.log 2024-04-24T01:24:59.036 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85554.log.gz 2024-04-24T01:24:59.036 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93408.log 2024-04-24T01:24:59.036 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40317.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115868.log.gz 2024-04-24T01:24:59.037 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86079.log.gz 2024-04-24T01:24:59.037 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82984.log 2024-04-24T01:24:59.037 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63106.log 2024-04-24T01:24:59.038 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31298.log 2024-04-24T01:24:59.038 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63736.log 2024-04-24T01:24:59.039 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63106.log: /var/log/ceph/cluster2-client.mirror.31298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85139.log 2024-04-24T01:24:59.039 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129788.log 2024-04-24T01:24:59.039 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.85139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135937.log 2024-04-24T01:24:59.040 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76435.log 2024-04-24T01:24:59.040 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135937.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129788.log.gz 2024-04-24T01:24:59.040 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135937.log.gz 2024-04-24T01:24:59.040 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33216.log 2024-04-24T01:24:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85108.log 2024-04-24T01:24:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88161.log 2024-04-24T01:24:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.85108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135644.log 2024-04-24T01:24:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59763.log 2024-04-24T01:24:59.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118404.log 2024-04-24T01:24:59.042 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.135644.log.gz 2024-04-24T01:24:59.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39638.log 2024-04-24T01:24:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118404.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118404.log.gz 2024-04-24T01:24:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48097.log 2024-04-24T01:24:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127315.log 2024-04-24T01:24:59.044 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107217.log 2024-04-24T01:24:59.044 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127315.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33348.log 2024-04-24T01:24:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107217.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127315.log.gz 2024-04-24T01:24:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36008.log 2024-04-24T01:24:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137526.log 2024-04-24T01:24:59.045 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107217.log.gz 2024-04-24T01:24:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36008.log: /var/log/ceph/cluster2-client.mirror.33348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97740.log 2024-04-24T01:24:59.047 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.137526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110898.log 2024-04-24T01:24:59.047 INFO:teuthology.orchestra.run.smithi134.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.137526.log.gz 2024-04-24T01:24:59.047 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76995.log 2024-04-24T01:24:59.047 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105710.log 2024-04-24T01:24:59.047 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76995.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109164.log.gz 2024-04-24T01:24:59.048 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102805.log 2024-04-24T01:24:59.048 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105710.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126694.log 2024-04-24T01:24:59.048 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.102805.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.102805.log.gz 2024-04-24T01:24:59.049 INFO:teuthology.orchestra.run.smithi134.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110898.log.gz 2024-04-24T01:24:59.049 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106804.log 2024-04-24T01:24:59.049 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60394.log.gz 2024-04-24T01:24:59.049 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126694.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60779.log.gz 2024-04-24T01:24:59.049 INFO:teuthology.orchestra.run.smithi134.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105710.log.gz 2024-04-24T01:24:59.049 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60989.log.gz 2024-04-24T01:24:59.050 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108061.log 2024-04-24T01:24:59.051 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106804.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93303.log 2024-04-24T01:24:59.051 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108061.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106804.log.gz 2024-04-24T01:24:59.051 INFO:teuthology.orchestra.run.smithi134.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126694.log.gz 2024-04-24T01:24:59.051 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108061.log.gz 2024-04-24T01:24:59.053 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69301.log 2024-04-24T01:24:59.053 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39832.log 2024-04-24T01:24:59.053 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110564.log 2024-04-24T01:24:59.053 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66357.log.gz 2024-04-24T01:24:59.053 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40608.log 2024-04-24T01:24:59.053 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138870.log 2024-04-24T01:24:59.053 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110564.log: /var/log/ceph/cluster2-client.mirror.40608.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134290.log 2024-04-24T01:24:59.054 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88336.log 2024-04-24T01:24:59.054 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110564.log.gz 2024-04-24T01:24:59.054 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133455.log 2024-04-24T01:24:59.054 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88336.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.138870.log.gz 2024-04-24T01:24:59.054 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47255.log 2024-04-24T01:24:59.055 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133455.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134290.log.gz 2024-04-24T01:24:59.055 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33906.log 2024-04-24T01:24:59.055 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133455.log.gz 2024-04-24T01:24:59.056 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62875.log 2024-04-24T01:24:59.056 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86926.log 2024-04-24T01:24:59.057 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94003.log 2024-04-24T01:24:59.057 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.86926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139296.log 2024-04-24T01:24:59.057 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34243.log.gz 2024-04-24T01:24:59.058 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34336.log.gz 2024-04-24T01:24:59.058 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79379.log 2024-04-24T01:24:59.058 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134199.log 2024-04-24T01:24:59.058 INFO:teuthology.orchestra.run.smithi134.stderr: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139296.log.gz 2024-04-24T01:24:59.058 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79379.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40317.log.gz 2024-04-24T01:24:59.058 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104110.log 2024-04-24T01:24:59.060 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134199.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53562.log.gz 2024-04-24T01:24:59.060 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45321.log 2024-04-24T01:24:59.060 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134199.log.gz 2024-04-24T01:24:59.060 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104110.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75899.log 2024-04-24T01:24:59.060 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45321.log: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.29749.log 2024-04-24T01:24:59.061 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.104110.log.gz 2024-04-24T01:24:59.061 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100113.log 2024-04-24T01:24:59.061 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.29749.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39541.log.gz 2024-04-24T01:24:59.061 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39638.log.gz 2024-04-24T01:24:59.061 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31577.log 2024-04-24T01:24:59.064 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39735.log.gz 2024-04-24T01:24:59.064 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100113.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100113.log.gz 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40088.log.gz 2024-04-24T01:24:59.064 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:59.064 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52297.log 2024-04-24T01:24:59.064 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40123.log.gz 2024-04-24T01:24:59.064 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40185.log.gz 2024-04-24T01:24:59.064 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133952.log 2024-04-24T01:24:59.064 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75789.log 2024-04-24T01:24:59.064 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52297.log: /var/log/ceph/cluster2-client.mirror.133952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138264.log 2024-04-24T01:24:59.065 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40705.log.gz 2024-04-24T01:24:59.065 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.75789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.29799.log 2024-04-24T01:24:59.065 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133952.log.gz 2024-04-24T01:24:59.065 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44351.log 2024-04-24T01:24:59.065 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138264.log: /var/log/ceph/cluster2-client.mirror.2.29799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40573.log 2024-04-24T01:24:59.065 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44351.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41481.log.gz 2024-04-24T01:24:59.068 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86429.log 2024-04-24T01:24:59.068 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138264.log.gz 2024-04-24T01:24:59.068 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44122.log.gz 2024-04-24T01:24:59.068 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46097.log 2024-04-24T01:24:59.068 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132810.log 2024-04-24T01:24:59.068 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31577.log.gz 2024-04-24T01:24:59.068 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43284.log 2024-04-24T01:24:59.068 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132810.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118028.log 2024-04-24T01:24:59.068 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43284.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136769.log 2024-04-24T01:24:59.069 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.132810.log.gz 2024-04-24T01:24:59.069 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118028.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62700.log.gz 2024-04-24T01:24:59.069 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60359.log 2024-04-24T01:24:59.069 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118028.log.gz 2024-04-24T01:24:59.069 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136769.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136769.log.gz 2024-04-24T01:24:59.069 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84798.log 2024-04-24T01:24:59.071 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110377.log 2024-04-24T01:24:59.071 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63106.log.gz 2024-04-24T01:24:59.071 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60359.log: /var/log/ceph/cluster2-client.mirror.84798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132245.log 2024-04-24T01:24:59.071 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.110377.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86991.log 2024-04-24T01:24:59.071 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43540.log 2024-04-24T01:24:59.071 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110377.log.gz 2024-04-24T01:24:59.072 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86991.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132245.log.gz 2024-04-24T01:24:59.072 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139619.log 2024-04-24T01:24:59.072 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30140.log 2024-04-24T01:24:59.072 INFO:teuthology.orchestra.run.smithi134.stderr: 95.5%/var/log/ceph/cluster2-client.mirror.43540.log: -- replaced with /var/log/ceph/cluster2-client.mirror.99962.log.gz 2024-04-24T01:24:59.072 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79939.log 2024-04-24T01:24:59.072 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139619.log.gz 2024-04-24T01:24:59.072 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.30140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60149.log 2024-04-24T01:24:59.072 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138985.log 2024-04-24T01:24:59.073 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130382.log 2024-04-24T01:24:59.073 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60149.log: /var/log/ceph/cluster2-client.mirror.138985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-04-24T01:24:59.073 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130382.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95304.log.gz 2024-04-24T01:24:59.073 INFO:teuthology.orchestra.run.smithi134.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.138985.log.gz 2024-04-24T01:24:59.074 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90898.log 2024-04-24T01:24:59.075 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100968.log 2024-04-24T01:24:59.075 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125520.log 2024-04-24T01:24:59.075 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100381.log 2024-04-24T01:24:59.076 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33685.log 2024-04-24T01:24:59.076 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100381.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100381.log.gz 2024-04-24T01:24:59.079 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69301.log.gz 2024-04-24T01:24:59.079 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69476.log.gz 2024-04-24T01:24:59.080 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69721.log.gz 2024-04-24T01:24:59.080 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69931.log.gz 2024-04-24T01:24:59.081 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52297.log.gz 2024-04-24T01:24:59.082 INFO:teuthology.orchestra.run.smithi134.stderr: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125520.log.gz 2024-04-24T01:24:59.083 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140985.log 2024-04-24T01:24:59.083 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34371.log 2024-04-24T01:24:59.084 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35909.log 2024-04-24T01:24:59.084 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34371.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46735.log 2024-04-24T01:24:59.084 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.140985.log.gz 2024-04-24T01:24:59.086 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.35909.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97192.log.gz 2024-04-24T01:24:59.086 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82529.log 2024-04-24T01:24:59.086 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46735.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77940.log 2024-04-24T01:24:59.086 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39894.log 2024-04-24T01:24:59.087 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82529.log: /var/log/ceph/cluster2-client.mirror.77940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132194.log 2024-04-24T01:24:59.087 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39894.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46097.log.gz 2024-04-24T01:24:59.091 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131394.log 2024-04-24T01:24:59.091 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61199.log 2024-04-24T01:24:59.091 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.131394.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.131394.log.gz 2024-04-24T01:24:59.092 INFO:teuthology.orchestra.run.smithi134.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132194.log.gz 2024-04-24T01:24:59.093 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81061.log 2024-04-24T01:24:59.093 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34057.log 2024-04-24T01:24:59.094 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37015.log 2024-04-24T01:24:59.094 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111944.log 2024-04-24T01:24:59.095 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130773.log 2024-04-24T01:24:59.095 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111944.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100352.log 2024-04-24T01:24:59.095 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130773.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130773.log.gz 2024-04-24T01:24:59.096 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33216.log.gz 2024-04-24T01:24:59.097 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33348.log.gz 2024-04-24T01:24:59.098 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.85108.log.gz 2024-04-24T01:24:59.099 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36008.log.gz 2024-04-24T01:24:59.099 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72651.log 2024-04-24T01:24:59.099 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36322.log.gz 2024-04-24T01:24:59.100 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127280.log 2024-04-24T01:24:59.101 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72651.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38571.log.gz 2024-04-24T01:24:59.101 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38633.log.gz 2024-04-24T01:24:59.102 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86814.log.gz 2024-04-24T01:24:59.102 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90023.log.gz 2024-04-24T01:24:59.102 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.90058.log.gz 2024-04-24T01:24:59.102 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85799.log.gz 2024-04-24T01:24:59.103 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134248.log 2024-04-24T01:24:59.103 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127280.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.127280.log.gz 2024-04-24T01:24:59.103 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81233.log 2024-04-24T01:24:59.104 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134248.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86429.log.gz 2024-04-24T01:24:59.105 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134248.log.gz 2024-04-24T01:24:59.106 INFO:teuthology.orchestra.run.smithi134.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101340.log.gz 2024-04-24T01:24:59.106 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127816.log 2024-04-24T01:24:59.107 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81233.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136454.log 2024-04-24T01:24:59.108 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77065.log 2024-04-24T01:24:59.108 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127816.log: /var/log/ceph/cluster1-client.mirror.136454.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136454.log.gz 2024-04-24T01:24:59.108 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127816.log.gz 2024-04-24T01:24:59.108 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107964.log 2024-04-24T01:24:59.109 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128969.log 2024-04-24T01:24:59.109 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77065.log: /var/log/ceph/cluster2-client.mirror.107964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42164.log 2024-04-24T01:24:59.109 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100048.log.gz 2024-04-24T01:24:59.110 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.107964.log.gz 2024-04-24T01:24:59.110 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.128969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95433.log 2024-04-24T01:24:59.111 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55273.log 2024-04-24T01:24:59.111 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42164.log: /var/log/ceph/cluster1-client.mirror.95433.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.128969.log.gz 2024-04-24T01:24:59.111 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100243.log 2024-04-24T01:24:59.112 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139769.log 2024-04-24T01:24:59.112 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133426.log 2024-04-24T01:24:59.112 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100243.log: /var/log/ceph/cluster2-client.mirror.139769.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100243.log.gz 2024-04-24T01:24:59.112 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124847.log 2024-04-24T01:24:59.113 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135143.log 2024-04-24T01:24:59.113 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133426.log: /var/log/ceph/cluster2-client.mirror.124847.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124847.log.gz 2024-04-24T01:24:59.114 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35173.log 2024-04-24T01:24:59.115 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.135143.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133426.log.gz 2024-04-24T01:24:59.115 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135143.log.gz 2024-04-24T01:24:59.115 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94095.log 2024-04-24T01:24:59.115 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33406.log 2024-04-24T01:24:59.116 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136859.log 2024-04-24T01:24:59.116 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79379.log.gz 2024-04-24T01:24:59.117 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75922.log 2024-04-24T01:24:59.117 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79519.log.gz 2024-04-24T01:24:59.117 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99706.log 2024-04-24T01:24:59.117 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136859.log.gz 2024-04-24T01:24:59.118 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133384.log 2024-04-24T01:24:59.118 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99706.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99706.log.gz 2024-04-24T01:24:59.119 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132010.log 2024-04-24T01:24:59.120 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133384.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133384.log.gz 2024-04-24T01:24:59.120 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67796.log 2024-04-24T01:24:59.122 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132010.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94138.log.gz 2024-04-24T01:24:59.123 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132010.log.gz 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72511.log.gz 2024-04-24T01:24:59.123 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:59.123 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70530.log 2024-04-24T01:24:59.123 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60184.log 2024-04-24T01:24:59.124 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.121039.log 2024-04-24T01:24:59.126 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70530.log: /var/log/ceph/cluster2-client.mirror.60184.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70565.log 2024-04-24T01:24:59.127 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.121039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45771.log 2024-04-24T01:24:59.128 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38924.log 2024-04-24T01:24:59.129 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58396.log 2024-04-24T01:24:59.129 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105652.log 2024-04-24T01:24:59.131 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84693.log 2024-04-24T01:24:59.132 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105652.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105652.log.gz 2024-04-24T01:24:59.132 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131578.log 2024-04-24T01:24:59.133 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41446.log 2024-04-24T01:24:59.135 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131578.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67376.log.gz 2024-04-24T01:24:59.135 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.131578.log.gz 2024-04-24T01:24:59.136 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70775.log 2024-04-24T01:24:59.137 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137377.log 2024-04-24T01:24:59.137 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113231.log 2024-04-24T01:24:59.138 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137377.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33685.log.gz 2024-04-24T01:24:59.138 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137377.log.gz 2024-04-24T01:24:59.139 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126509.log 2024-04-24T01:24:59.139 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67796.log.gz 2024-04-24T01:24:59.139 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.113231.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88161.log.gz 2024-04-24T01:24:59.140 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88336.log.gz 2024-04-24T01:24:59.141 INFO:teuthology.orchestra.run.smithi134.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113231.log.gz 2024-04-24T01:24:59.141 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132688.log 2024-04-24T01:24:59.142 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39832.log.gz 2024-04-24T01:24:59.142 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39894.log.gz 2024-04-24T01:24:59.142 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126509.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40608.log.gz 2024-04-24T01:24:59.143 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41446.log.gz 2024-04-24T01:24:59.143 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46735.log.gz 2024-04-24T01:24:59.143 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126509.log.gz 2024-04-24T01:24:59.143 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67936.log 2024-04-24T01:24:59.145 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132688.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132688.log.gz 2024-04-24T01:24:59.145 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118196.log 2024-04-24T01:24:59.146 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67936.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39444.log 2024-04-24T01:24:59.147 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118196.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118196.log.gz 2024-04-24T01:24:59.147 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70141.log 2024-04-24T01:24:59.148 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29248.log 2024-04-24T01:24:59.149 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43963.log 2024-04-24T01:24:59.150 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29248.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29248.log.gz 2024-04-24T01:24:59.150 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128892.log 2024-04-24T01:24:59.151 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42954.log 2024-04-24T01:24:59.152 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128892.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128892.log.gz 2024-04-24T01:24:59.152 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124245.log 2024-04-24T01:24:59.153 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128327.log 2024-04-24T01:24:59.155 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.124245.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124245.log.gz 2024-04-24T01:24:59.156 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61199.log.gz 2024-04-24T01:24:59.157 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72651.log.gz 2024-04-24T01:24:59.157 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72721.log.gz 2024-04-24T01:24:59.157 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51486.log 2024-04-24T01:24:59.159 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128327.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.121039.log.gz 2024-04-24T01:24:59.159 INFO:teuthology.orchestra.run.smithi134.stderr: 92.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103498.log 2024-04-24T01:24:59.159 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.128327.log.gz 2024-04-24T01:24:59.160 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51486.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128485.log 2024-04-24T01:24:59.160 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42196.log 2024-04-24T01:24:59.161 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128485.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80728.log.gz 2024-04-24T01:24:59.161 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128485.log.gz 2024-04-24T01:24:59.163 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76785.log.gz 2024-04-24T01:24:59.164 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31298.log.gz 2024-04-24T01:24:59.165 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92953.log.gz 2024-04-24T01:24:59.165 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43284.log.gz 2024-04-24T01:24:59.166 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58396.log.gz 2024-04-24T01:24:59.166 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62875.log.gz 2024-04-24T01:24:59.169 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91108.log.gz 2024-04-24T01:24:59.169 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36043.log 2024-04-24T01:24:59.170 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.42196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121507.log 2024-04-24T01:24:59.170 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51486.log.gz 2024-04-24T01:24:59.172 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36043.log: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30140.log.gz 2024-04-24T01:24:59.175 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59833.log.gz 2024-04-24T01:24:59.175 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59868.log.gz 2024-04-24T01:24:59.175 INFO:teuthology.orchestra.run.smithi134.stderr: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30367.log.gz 2024-04-24T01:24:59.176 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45771.log.gz 2024-04-24T01:24:59.177 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33406.log.gz 2024-04-24T01:24:59.177 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86604.log.gz 2024-04-24T01:24:59.177 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131507.log 2024-04-24T01:24:59.179 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121507.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121507.log.gz 2024-04-24T01:24:59.179 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43051.log 2024-04-24T01:24:59.181 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131507.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131507.log.gz 2024-04-24T01:24:59.182 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49775.log 2024-04-24T01:24:59.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58493.log 2024-04-24T01:24:59.188 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49775.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82529.log.gz 2024-04-24T01:24:59.189 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82984.log.gz 2024-04-24T01:24:59.189 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82844.log.gz 2024-04-24T01:24:59.194 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130382.log.gz 2024-04-24T01:24:59.194 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139160.log 2024-04-24T01:24:59.195 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135577.log 2024-04-24T01:24:59.195 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134270.log 2024-04-24T01:24:59.196 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139160.log: /var/log/ceph/cluster1-client.mirror.135577.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135577.log.gz 2024-04-24T01:24:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100025.log 2024-04-24T01:24:59.197 INFO:teuthology.orchestra.run.smithi134.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.139160.log.gz 2024-04-24T01:24:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134270.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134270.log.gz 2024-04-24T01:24:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-04-24T01:24:59.198 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79694.log 2024-04-24T01:24:59.198 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100025.log: /var/log/ceph/cluster2-mon.a.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100025.log.gz 2024-04-24T01:24:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71720.log 2024-04-24T01:24:59.199 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79939.log.gz 2024-04-24T01:24:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54920.log 2024-04-24T01:24:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32414.log 2024-04-24T01:24:59.201 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135779.log 2024-04-24T01:24:59.210 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54920.log: /var/log/ceph/cluster2-client.mirror.32414.log: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100968.log.gz 2024-04-24T01:24:59.213 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47546.log.gz 2024-04-24T01:24:59.213 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117931.log 2024-04-24T01:24:59.214 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106232.log 2024-04-24T01:24:59.215 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8%/var/log/ceph/cluster2-client.mirror.117931.log: -- replaced with /var/log/ceph/cluster1-client.mirror.135779.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82144.log 2024-04-24T01:24:59.215 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:59.215 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117931.log.gz 2024-04-24T01:24:59.215 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64486.log 2024-04-24T01:24:59.216 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106232.log: /var/log/ceph/cluster2-client.mirror.82144.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55696.log 2024-04-24T01:24:59.216 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60184.log.gz 2024-04-24T01:24:59.216 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60149.log.gz 2024-04-24T01:24:59.216 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64486.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60359.log.gz 2024-04-24T01:24:59.216 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122410.log 2024-04-24T01:24:59.217 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35673.log 2024-04-24T01:24:59.218 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59763.log.gz 2024-04-24T01:24:59.218 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135262.log 2024-04-24T01:24:59.218 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122410.log: /var/log/ceph/cluster2-client.mirror.35673.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70775.log.gz 2024-04-24T01:24:59.218 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122410.log.gz 2024-04-24T01:24:59.218 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80569.log 2024-04-24T01:24:59.218 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68041.log 2024-04-24T01:24:59.218 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80569.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135262.log.gz 2024-04-24T01:24:59.218 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71720.log.gz 2024-04-24T01:24:59.219 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141244.log 2024-04-24T01:24:59.219 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54532.log 2024-04-24T01:24:59.219 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68041.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.90898.log.gz 2024-04-24T01:24:59.219 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67936.log.gz 2024-04-24T01:24:59.219 INFO:teuthology.orchestra.run.smithi134.stderr: 74.3% -- replaced with /var/log/ceph/cluster1-client.mirror.81233.log.gz 2024-04-24T01:24:59.219 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132387.log 2024-04-24T01:24:59.222 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141244.log: /var/log/ceph/cluster2-client.mirror.54532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129239.log 2024-04-24T01:24:59.223 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132387.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132387.log.gz 2024-04-24T01:24:59.224 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58202.log 2024-04-24T01:24:59.225 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129239.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129239.log.gz 2024-04-24T01:24:59.225 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75877.log 2024-04-24T01:24:59.226 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115411.log 2024-04-24T01:24:59.226 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.75877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42067.log 2024-04-24T01:24:59.227 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115411.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115411.log.gz 2024-04-24T01:24:59.229 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33906.log.gz 2024-04-24T01:24:59.230 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34057.log.gz 2024-04-24T01:24:59.230 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108558.log 2024-04-24T01:24:59.230 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34371.log.gz 2024-04-24T01:24:59.231 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37077.log 2024-04-24T01:24:59.231 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108558.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108558.log.gz 2024-04-24T01:24:59.232 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30401.log 2024-04-24T01:24:59.232 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39444.log.gz 2024-04-24T01:24:59.233 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40573.log.gz 2024-04-24T01:24:59.233 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42067.log.gz 2024-04-24T01:24:59.234 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68986.log 2024-04-24T01:24:59.234 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42164.log.gz 2024-04-24T01:24:59.235 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.30401.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42196.log.gz 2024-04-24T01:24:59.235 INFO:teuthology.orchestra.run.smithi134.stderr:gzip 88.2% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.44351.log.gz --verbose 2024-04-24T01:24:59.235 INFO:teuthology.orchestra.run.smithi134.stderr: -- /var/log/ceph/cluster2-client.mirror.141156.log 2024-04-24T01:24:59.236 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68986.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47255.log.gz 2024-04-24T01:24:59.236 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80569.log.gz 2024-04-24T01:24:59.236 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113026.log 2024-04-24T01:24:59.238 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141156.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141156.log.gz 2024-04-24T01:24:59.238 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61619.log 2024-04-24T01:24:59.239 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113026.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113026.log.gz 2024-04-24T01:24:59.239 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124581.log 2024-04-24T01:24:59.240 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40026.log 2024-04-24T01:24:59.240 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124581.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124581.log.gz 2024-04-24T01:24:59.241 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58264.log 2024-04-24T01:24:59.241 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42954.log.gz 2024-04-24T01:24:59.242 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40026.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43051.log.gz 2024-04-24T01:24:59.242 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29651.log 2024-04-24T01:24:59.242 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96076.log 2024-04-24T01:24:59.243 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.29651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52491.log 2024-04-24T01:24:59.244 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96076.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35673.log.gz 2024-04-24T01:24:59.245 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58493.log.gz 2024-04-24T01:24:59.246 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63736.log.gz 2024-04-24T01:24:59.246 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68986.log.gz 2024-04-24T01:24:59.247 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41873.log 2024-04-24T01:24:59.247 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93373.log 2024-04-24T01:24:59.249 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94956.log.gz 2024-04-24T01:24:59.250 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36043.log.gz 2024-04-24T01:24:59.250 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136949.log 2024-04-24T01:24:59.251 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93373.log: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100352.log.gz 2024-04-24T01:24:59.252 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60254.log 2024-04-24T01:24:59.252 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38924.log.gz 2024-04-24T01:24:59.252 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45321.log.gz 2024-04-24T01:24:59.252 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136949.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.136949.log.gz 2024-04-24T01:24:59.253 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52491.log.gz 2024-04-24T01:24:59.253 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134045.log 2024-04-24T01:24:59.253 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81083.log 2024-04-24T01:24:59.254 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131487.log 2024-04-24T01:24:59.254 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134045.log: /var/log/ceph/cluster2-client.mirror.81083.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82739.log 2024-04-24T01:24:59.255 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.131487.log: -- replaced with /var/log/ceph/cluster2-client.mirror.134045.log.gz 2024-04-24T01:24:59.255 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109466.log 2024-04-24T01:24:59.256 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.131487.log.gz 2024-04-24T01:24:59.256 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107778.log 2024-04-24T01:24:59.257 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84553.log.gz 2024-04-24T01:24:59.257 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109466.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84693.log.gz 2024-04-24T01:24:59.257 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84798.log.gz 2024-04-24T01:24:59.257 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109466.log.gz 2024-04-24T01:24:59.258 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64508.log 2024-04-24T01:24:59.258 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51777.log 2024-04-24T01:24:59.258 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107778.log: /var/log/ceph/cluster2-client.mirror.64508.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93303.log.gz 2024-04-24T01:24:59.258 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107778.log.gz 2024-04-24T01:24:59.258 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93408.log.gz 2024-04-24T01:24:59.259 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109727.log 2024-04-24T01:24:59.259 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53236.log 2024-04-24T01:24:59.259 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55273.log.gz 2024-04-24T01:24:59.260 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51777.log: /var/log/ceph/cluster1-client.mirror.109727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131116.log 2024-04-24T01:24:59.260 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55696.log.gz 2024-04-24T01:24:59.260 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.109727.log.gz 2024-04-24T01:24:59.261 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53236.log: 64.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30401.log.gz 2024-04-24T01:24:59.261 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49775.log.gz 2024-04-24T01:24:59.262 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37077.log.gz 2024-04-24T01:24:59.263 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.85139.log.gz 2024-04-24T01:24:59.267 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24288.log 2024-04-24T01:24:59.267 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113253.log 2024-04-24T01:24:59.268 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.24288.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131116.log.gz 2024-04-24T01:24:59.269 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68146.log 2024-04-24T01:24:59.269 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55890.log 2024-04-24T01:24:59.269 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113253.log: /var/log/ceph/cluster2-client.mirror.68146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140554.log 2024-04-24T01:24:59.270 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136297.log 2024-04-24T01:24:59.271 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140554.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140554.log.gz 2024-04-24T01:24:59.271 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64121.log 2024-04-24T01:24:59.272 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136297.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136297.log.gz 2024-04-24T01:24:59.272 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99247.log 2024-04-24T01:24:59.274 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36980.log 2024-04-24T01:24:59.275 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99247.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99247.log.gz 2024-04-24T01:24:59.275 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28932.log 2024-04-24T01:24:59.277 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56994.log 2024-04-24T01:24:59.277 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.28932.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28932.log.gz 2024-04-24T01:24:59.277 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123569.log 2024-04-24T01:24:59.277 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135959.log 2024-04-24T01:24:59.280 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123569.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.123569.log.gz 2024-04-24T01:24:59.280 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76855.log 2024-04-24T01:24:59.281 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135959.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135959.log.gz 2024-04-24T01:24:59.281 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122445.log 2024-04-24T01:24:59.281 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87216.log 2024-04-24T01:24:59.282 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122445.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122445.log.gz 2024-04-24T01:24:59.283 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124658.log 2024-04-24T01:24:59.283 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38086.log 2024-04-24T01:24:59.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124658.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124658.log.gz 2024-04-24T01:24:59.284 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133406.log 2024-04-24T01:24:59.285 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55017.log 2024-04-24T01:24:59.286 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133406.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133406.log.gz 2024-04-24T01:24:59.286 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111000.log 2024-04-24T01:24:59.293 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55017.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82144.log.gz 2024-04-24T01:24:59.293 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82739.log.gz 2024-04-24T01:24:59.293 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77940.log.gz 2024-04-24T01:24:59.294 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80359.log 2024-04-24T01:24:59.294 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111000.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61619.log.gz 2024-04-24T01:24:59.295 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111000.log.gz 2024-04-24T01:24:59.295 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29018.log 2024-04-24T01:24:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37663.log 2024-04-24T01:24:59.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29018.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29018.log.gz 2024-04-24T01:24:59.297 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95589.log 2024-04-24T01:24:59.298 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52689.log 2024-04-24T01:24:59.298 INFO:teuthology.orchestra.run.smithi134.stderr: 83.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35909.log.gz 2024-04-24T01:24:59.299 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95589.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40026.log.gz 2024-04-24T01:24:59.299 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41873.log.gz 2024-04-24T01:24:59.300 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53236.log.gz 2024-04-24T01:24:59.302 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50807.log 2024-04-24T01:24:59.304 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52689.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48097.log.gz 2024-04-24T01:24:59.304 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60254.log.gz 2024-04-24T01:24:59.305 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93583.log 2024-04-24T01:24:59.305 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42284.log 2024-04-24T01:24:59.306 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47643.log 2024-04-24T01:24:59.307 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.42284.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.93373.log.gz 2024-04-24T01:24:59.307 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124267.log 2024-04-24T01:24:59.307 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75789.log.gz 2024-04-24T01:24:59.308 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122983.log 2024-04-24T01:24:59.308 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35173.log.gz 2024-04-24T01:24:59.308 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32414.log.gz 2024-04-24T01:24:59.309 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124267.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50807.log.gz 2024-04-24T01:24:59.309 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58202.log.gz 2024-04-24T01:24:59.310 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58264.log.gz 2024-04-24T01:24:59.312 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43963.log.gz 2024-04-24T01:24:59.316 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43540.log.gz 2024-04-24T01:24:59.318 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64121.log.gz 2024-04-24T01:24:59.327 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136319.log 2024-04-24T01:24:59.333 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122983.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96076.log.gz 2024-04-24T01:24:59.338 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52689.log.gz 2024-04-24T01:24:59.339 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128857.log 2024-04-24T01:24:59.339 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136319.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136319.log.gz 2024-04-24T01:24:59.354 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55017.log.gz 2024-04-24T01:24:59.357 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55890.log.gz 2024-04-24T01:24:59.357 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54532.log.gz 2024-04-24T01:24:59.357 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97584.log 2024-04-24T01:24:59.358 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128857.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36980.log.gz 2024-04-24T01:24:59.359 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128857.log.gz 2024-04-24T01:24:59.359 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37015.log.gz 2024-04-24T01:24:59.359 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.93583.log.gz 2024-04-24T01:24:59.370 INFO:teuthology.orchestra.run.smithi134.stderr: 28.9% -- replaced with /var/log/ceph/cluster2-client.admin.24288.log.gz 2024-04-24T01:24:59.370 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95566.log 2024-04-24T01:24:59.372 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97584.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111944.log.gz 2024-04-24T01:24:59.372 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51777.log.gz 2024-04-24T01:24:59.373 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54920.log.gz 2024-04-24T01:24:59.375 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56994.log.gz 2024-04-24T01:24:59.383 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79694.log.gz 2024-04-24T01:24:59.390 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38086.log.gz 2024-04-24T01:24:59.391 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102979.log 2024-04-24T01:24:59.410 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135689.log 2024-04-24T01:24:59.411 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.102979.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.102979.log.gz 2024-04-24T01:24:59.414 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95433.log.gz 2024-04-24T01:24:59.423 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95456.log.gz 2024-04-24T01:24:59.431 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65216.log 2024-04-24T01:24:59.431 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135689.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135689.log.gz 2024-04-24T01:24:59.433 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.94003.log.gz 2024-04-24T01:24:59.439 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75877.log.gz 2024-04-24T01:24:59.447 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41287.log 2024-04-24T01:24:59.447 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103498.log.gz 2024-04-24T01:24:59.465 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65216.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139769.log.gz 2024-04-24T01:24:59.467 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37663.log.gz 2024-04-24T01:24:59.468 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133690.log 2024-04-24T01:24:59.469 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141136.log 2024-04-24T01:24:59.469 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.113253.log.gz 2024-04-24T01:24:59.469 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133690.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87216.log.gz 2024-04-24T01:24:59.469 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133690.log.gz 2024-04-24T01:24:59.469 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47643.log.gz 2024-04-24T01:24:59.469 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97260.log 2024-04-24T01:24:59.470 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141136.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32751.log 2024-04-24T01:24:59.471 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141136.log.gz 2024-04-24T01:24:59.471 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.97260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30435.log 2024-04-24T01:24:59.471 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44157.log 2024-04-24T01:24:59.471 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126019.log 2024-04-24T01:24:59.472 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.30435.log: /var/log/ceph/cluster2-client.mirror.44157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87050.log 2024-04-24T01:24:59.472 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38730.log 2024-04-24T01:24:59.473 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.126019.log.gz 2024-04-24T01:24:59.473 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.87050.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109398.log 2024-04-24T01:24:59.473 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49096.log 2024-04-24T01:24:59.474 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38730.log: /var/log/ceph/cluster2-client.mirror.109398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111922.log 2024-04-24T01:24:59.474 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101099.log 2024-04-24T01:24:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.111922.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111922.log.gz 2024-04-24T01:24:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50842.log 2024-04-24T01:24:59.478 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101099.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101099.log.gz 2024-04-24T01:24:59.478 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104737.log 2024-04-24T01:24:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113519.log 2024-04-24T01:24:59.485 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104737.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32751.log.gz 2024-04-24T01:24:59.485 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50842.log.gz 2024-04-24T01:24:59.486 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93898.log 2024-04-24T01:24:59.487 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113519.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113519.log.gz 2024-04-24T01:24:59.490 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80359.log.gz 2024-04-24T01:24:59.495 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105035.log 2024-04-24T01:24:59.495 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93898.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38730.log.gz 2024-04-24T01:24:59.502 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% 55.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30435.log.gz 2024-04-24T01:24:59.502 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.106232.log.gz 2024-04-24T01:24:59.502 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52200.log 2024-04-24T01:24:59.502 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49096.log.gz 2024-04-24T01:24:59.504 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105035.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.93898.log.gz 2024-04-24T01:24:59.505 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95589.log.gz 2024-04-24T01:24:59.506 INFO:teuthology.orchestra.run.smithi134.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97260.log.gz 2024-04-24T01:24:59.506 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105035.log.gz 2024-04-24T01:24:59.507 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55211.log 2024-04-24T01:24:59.507 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31356.log 2024-04-24T01:24:59.507 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106335.log 2024-04-24T01:24:59.508 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55308.log 2024-04-24T01:24:59.509 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106335.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106335.log.gz 2024-04-24T01:24:59.509 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111259.log 2024-04-24T01:24:59.510 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84273.log 2024-04-24T01:24:59.513 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.111259.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111259.log.gz 2024-04-24T01:24:59.514 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75175.log 2024-04-24T01:24:59.515 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84273.log: 95.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137242.log 2024-04-24T01:24:59.515 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.141244.log.gz 2024-04-24T01:24:59.515 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32786.log 2024-04-24T01:24:59.516 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112200.log 2024-04-24T01:24:59.516 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32786.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137242.log.gz 2024-04-24T01:24:59.516 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86744.log 2024-04-24T01:24:59.516 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35022.log 2024-04-24T01:24:59.517 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86744.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112200.log.gz 2024-04-24T01:24:59.517 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96284.log 2024-04-24T01:24:59.518 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117966.log 2024-04-24T01:24:59.518 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35022.log: /var/log/ceph/cluster2-client.mirror.96284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.97260.log 2024-04-24T01:24:59.519 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113942.log 2024-04-24T01:24:59.519 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117966.log: /var/log/ceph/cluster2-client.mirror.0.97260.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117966.log.gz 2024-04-24T01:24:59.519 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106650.log 2024-04-24T01:24:59.520 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95089.log 2024-04-24T01:24:59.521 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.113942.log: /var/log/ceph/cluster1-client.mirror.106650.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113942.log.gz 87.9% 2024-04-24T01:24:59.521 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.106650.log.gz 2024-04-24T01:24:59.521 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64735.log 2024-04-24T01:24:59.521 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77205.log 2024-04-24T01:24:59.522 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95089.log: /var/log/ceph/cluster1-client.admin.64735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121761.log 2024-04-24T01:24:59.523 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82634.log 2024-04-24T01:24:59.523 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77205.log: /var/log/ceph/cluster1-client.mirror.121761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140021.log 2024-04-24T01:24:59.523 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121761.log.gz 2024-04-24T01:24:59.523 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105685.log 2024-04-24T01:24:59.524 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48479.log 2024-04-24T01:24:59.525 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140021.log: /var/log/ceph/cluster1-client.mirror.105685.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.140021.log.gz 2024-04-24T01:24:59.525 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105685.log.gz 2024-04-24T01:24:59.526 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94353.log 2024-04-24T01:24:59.527 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54497.log 2024-04-24T01:24:59.527 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48479.log: /var/log/ceph/cluster2-client.mirror.94353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114722.log 2024-04-24T01:24:59.528 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68216.log 2024-04-24T01:24:59.528 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.114722.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114722.log.gz 2024-04-24T01:24:59.529 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.122983.log.gz 2024-04-24T01:24:59.530 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106901.log 2024-04-24T01:24:59.533 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106300.log 2024-04-24T01:24:59.533 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64466.log 2024-04-24T01:24:59.534 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117837.log 2024-04-24T01:24:59.534 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106300.log.gz 2024-04-24T01:24:59.534 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135667.log 2024-04-24T01:24:59.535 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82634.log.gz 2024-04-24T01:24:59.536 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64466.log: /var/log/ceph/cluster2-client.mirror.117837.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106901.log.gz 2024-04-24T01:24:59.536 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.87050.log.gz 2024-04-24T01:24:59.537 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96344.log 2024-04-24T01:24:59.537 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135667.log: gzip 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117837.log.gz -5 2024-04-24T01:24:59.537 INFO:teuthology.orchestra.run.smithi134.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.132265.log 2024-04-24T01:24:59.538 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135667.log.gz 2024-04-24T01:24:59.538 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86219.log 2024-04-24T01:24:59.538 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96344.log: /var/log/ceph/cluster2-client.mirror.132265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138583.log 2024-04-24T01:24:59.538 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86219.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.64735.log.gz 2024-04-24T01:24:59.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41252.log 2024-04-24T01:24:59.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138583.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132265.log.gz 2024-04-24T01:24:59.539 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.138583.log.gz 2024-04-24T01:24:59.539 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77205.log.gz 2024-04-24T01:24:59.540 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41287.log.gz 2024-04-24T01:24:59.540 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44157.log.gz 2024-04-24T01:24:59.540 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.86926.log.gz 2024-04-24T01:24:59.541 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117879.log 2024-04-24T01:24:59.541 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94983.log 2024-04-24T01:24:59.542 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99623.log 2024-04-24T01:24:59.542 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117879.log: /var/log/ceph/cluster2-client.mirror.94983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59308.log 2024-04-24T01:24:59.543 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99623.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99623.log.gz 2024-04-24T01:24:59.543 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118496.log 2024-04-24T01:24:59.543 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59308.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84273.log.gz 2024-04-24T01:24:59.544 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36650.log 2024-04-24T01:24:59.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118496.log: 92.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78959.log 2024-04-24T01:24:59.546 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118496.log.gz 2024-04-24T01:24:59.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.36650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110918.log 2024-04-24T01:24:59.547 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112060.log 2024-04-24T01:24:59.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78959.log: /var/log/ceph/cluster1-client.mirror.110918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119292.log 2024-04-24T01:24:59.548 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112060.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112060.log.gz 2024-04-24T01:24:59.549 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28906.log 2024-04-24T01:24:59.549 INFO:teuthology.orchestra.run.smithi134.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110918.log.gz 2024-04-24T01:24:59.549 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119292.log: 88.0% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119292.log.gz 2024-04-24T01:24:59.550 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.31356.log.gz 2024-04-24T01:24:59.550 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32786.log.gz 2024-04-24T01:24:59.554 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64901.log 2024-04-24T01:24:59.555 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119140.log 2024-04-24T01:24:59.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.28906.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28906.log.gz 2024-04-24T01:24:59.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85274.log 2024-04-24T01:24:59.556 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75350.log 2024-04-24T01:24:59.556 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37310.log 2024-04-24T01:24:59.557 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75350.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65006.log 2024-04-24T01:24:59.562 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78325.log 2024-04-24T01:24:59.564 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132316.log 2024-04-24T01:24:59.565 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91283.log 2024-04-24T01:24:59.566 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132316.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132316.log.gz 2024-04-24T01:24:59.571 INFO:teuthology.orchestra.run.smithi134.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96284.log.gz 2024-04-24T01:24:59.571 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87706.log 2024-04-24T01:24:59.571 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47903.log 2024-04-24T01:24:59.571 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87706.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128679.log 2024-04-24T01:24:59.571 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47903.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95566.log.gz 2024-04-24T01:24:59.572 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.92368.log.gz 2024-04-24T01:24:59.572 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71055.log 2024-04-24T01:24:59.573 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128679.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128679.log.gz 2024-04-24T01:24:59.574 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97790.log 2024-04-24T01:24:59.574 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52200.log.gz 2024-04-24T01:24:59.574 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68741.log 2024-04-24T01:24:59.576 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97790.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29799.log.gz 2024-04-24T01:24:59.577 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76435.log.gz 2024-04-24T01:24:59.580 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77065.log.gz 2024-04-24T01:24:59.580 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76995.log.gz 2024-04-24T01:24:59.580 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55211.log.gz 2024-04-24T01:24:59.581 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55308.log.gz 2024-04-24T01:24:59.582 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54497.log.gz 2024-04-24T01:24:59.584 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37310.log.gz 2024-04-24T01:24:59.585 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86744.log.gz 2024-04-24T01:24:59.586 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65216.log.gz 2024-04-24T01:24:59.595 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68041.log.gz 2024-04-24T01:24:59.595 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68146.log.gz 2024-04-24T01:24:59.596 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76855.log.gz 2024-04-24T01:24:59.598 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.126773.log 2024-04-24T01:24:59.607 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68741.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109398.log.gz 2024-04-24T01:24:59.609 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61739.log 2024-04-24T01:24:59.609 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.2.126773.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70530.log.gz 2024-04-24T01:24:59.609 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70565.log.gz 2024-04-24T01:24:59.611 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78959.log.gz 2024-04-24T01:24:59.614 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124267.log.gz 2024-04-24T01:24:59.615 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138388.log 2024-04-24T01:24:59.615 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.61739.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68741.log.gz 2024-04-24T01:24:59.616 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67411.log 2024-04-24T01:24:59.616 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70141.log.gz 2024-04-24T01:24:59.617 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138388.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.138388.log.gz 2024-04-24T01:24:59.617 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94353.log.gz 2024-04-24T01:24:59.617 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134412.log 2024-04-24T01:24:59.618 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54047.log 2024-04-24T01:24:59.618 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134412.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134412.log.gz 2024-04-24T01:24:59.619 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133094.log 2024-04-24T01:24:59.620 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54047.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71055.log.gz 2024-04-24T01:24:59.620 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76750.log 2024-04-24T01:24:59.620 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78325.log.gz 2024-04-24T01:24:59.620 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81061.log.gz 2024-04-24T01:24:59.620 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133094.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133094.log.gz 2024-04-24T01:24:59.621 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79274.log 2024-04-24T01:24:59.621 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76750.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85274.log.gz 2024-04-24T01:24:59.621 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73456.log 2024-04-24T01:24:59.622 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79274.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59308.log.gz 2024-04-24T01:24:59.622 INFO:teuthology.orchestra.run.smithi134.stderr: 82.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61739.log.gz 2024-04-24T01:24:59.623 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131151.log 2024-04-24T01:24:59.623 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68216.log.gz 2024-04-24T01:24:59.623 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76750.log.gz 2024-04-24T01:24:59.623 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87706.log.gz 2024-04-24T01:24:59.624 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97132.log 2024-04-24T01:24:59.624 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131151.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.131151.log.gz 2024-04-24T01:24:59.625 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123877.log 2024-04-24T01:24:59.625 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89635.log 2024-04-24T01:24:59.626 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123877.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123877.log.gz 2024-04-24T01:24:59.626 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111136.log 2024-04-24T01:24:59.627 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73386.log 2024-04-24T01:24:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111136.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111136.log.gz 2024-04-24T01:24:59.629 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35022.log.gz 2024-04-24T01:24:59.629 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41252.log.gz 2024-04-24T01:24:59.630 INFO:teuthology.orchestra.run.smithi134.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.29749.log.gz 2024-04-24T01:24:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47903.log.gz 95.7% 2024-04-24T01:24:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119140.log.gz 2024-04-24T01:24:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73456.log.gz 2024-04-24T01:24:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64901.log.gz 2024-04-24T01:24:59.632 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.36650.log.gz 2024-04-24T01:24:59.633 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65006.log.gz 2024-04-24T01:24:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123231.log 2024-04-24T01:24:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34429.log 2024-04-24T01:24:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73386.log: /var/log/ceph/cluster2-client.mirror.123231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121248.log 2024-04-24T01:24:59.634 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123231.log.gz 2024-04-24T01:24:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45868.log 2024-04-24T01:24:59.634 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-04-24T01:24:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55079.log 2024-04-24T01:24:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121248.log: /var/log/ceph/cluster2-client.mirror.45868.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.121248.log.gz 2024-04-24T01:24:59.635 INFO:teuthology.orchestra.run.smithi134.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97132.log.gz 2024-04-24T01:24:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94248.log 2024-04-24T01:24:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.81183.log 2024-04-24T01:24:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64423.log 2024-04-24T01:24:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29348.log 2024-04-24T01:24:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.81183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115769.log 2024-04-24T01:24:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64423.log: /var/log/ceph/cluster2-client.admin.29348.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29348.log.gz 2024-04-24T01:24:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57304.log 2024-04-24T01:24:59.638 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67411.log.gz 2024-04-24T01:24:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115769.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115769.log.gz 2024-04-24T01:24:59.638 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48479.log.gz 2024-04-24T01:24:59.638 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55079.log.gz 2024-04-24T01:24:59.639 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45868.log.gz 2024-04-24T01:24:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70106.log 2024-04-24T01:24:59.639 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54047.log.gz 2024-04-24T01:24:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.57304.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75350.log.gz 2024-04-24T01:24:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65901.log 2024-04-24T01:24:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37989.log 2024-04-24T01:24:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65901.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79274.log.gz 2024-04-24T01:24:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29322.log 2024-04-24T01:24:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37989.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.81183.log.gz 2024-04-24T01:24:59.643 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122271.log 2024-04-24T01:24:59.648 INFO:teuthology.orchestra.run.smithi134.stderr: 97.0% -- replaced with /var/log/ceph/cluster2-client.mirror.86991.log.gz 2024-04-24T01:24:59.648 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29322.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29322.log.gz 2024-04-24T01:24:59.648 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135072.log 2024-04-24T01:24:59.648 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.122271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114009.log 2024-04-24T01:24:59.649 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.135072.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135072.log.gz 2024-04-24T01:24:59.651 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128714.log 2024-04-24T01:24:59.651 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122271.log.gz 2024-04-24T01:24:59.651 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114009.log: 94.1%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.96736.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47123.log 2024-04-24T01:24:59.651 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:59.652 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104505.log 2024-04-24T01:24:59.652 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128714.log.gz 2024-04-24T01:24:59.652 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44545.log 2024-04-24T01:24:59.653 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104505.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104505.log.gz 2024-04-24T01:24:59.653 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37989.log.gz 2024-04-24T01:24:59.653 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120363.log 2024-04-24T01:24:59.654 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122941.log 2024-04-24T01:24:59.654 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134978.log 2024-04-24T01:24:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.120363.log: /var/log/ceph/cluster2-client.mirror.122941.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.120363.log.gz 2024-04-24T01:24:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45965.log 2024-04-24T01:24:59.656 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134978.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.122941.log.gz 2024-04-24T01:24:59.656 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134978.log.gz 2024-04-24T01:24:59.656 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84448.log 2024-04-24T01:24:59.656 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92988.log 2024-04-24T01:24:59.657 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45965.log: /var/log/ceph/cluster2-client.mirror.84448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61654.log 2024-04-24T01:24:59.658 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142036.log 2024-04-24T01:24:59.658 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92988.log: /var/log/ceph/cluster2-client.mirror.61654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37372.log 2024-04-24T01:24:59.659 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99939.log 2024-04-24T01:24:59.659 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.142036.log: /var/log/ceph/cluster2-client.mirror.37372.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142036.log.gz 2024-04-24T01:24:59.659 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66462.log 2024-04-24T01:24:59.660 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136432.log 2024-04-24T01:24:59.660 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94248.log.gz 2024-04-24T01:24:59.660 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99939.log.gz 2024-04-24T01:24:59.662 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63211.log 2024-04-24T01:24:59.662 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140046.log 2024-04-24T01:24:59.662 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136432.log.gz 2024-04-24T01:24:59.663 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81106.log 2024-04-24T01:24:59.663 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49678.log 2024-04-24T01:24:59.664 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71855.log 2024-04-24T01:24:59.664 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61654.log.gz 2024-04-24T01:24:59.665 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49678.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140046.log.gz 2024-04-24T01:24:59.666 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111580.log 2024-04-24T01:24:59.666 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89635.log.gz 2024-04-24T01:24:59.666 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83248.log 2024-04-24T01:24:59.667 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111580.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111580.log.gz 2024-04-24T01:24:59.667 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119664.log 2024-04-24T01:24:59.667 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83786.log 2024-04-24T01:24:59.668 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.119664.log: /var/log/ceph/cluster1-client.mirror.130360.log 2024-04-24T01:24:59.668 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119664.log.gz/var/log/ceph/cluster2-client.mirror.83786.log: 2024-04-24T01:24:59.669 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31391.log 2024-04-24T01:24:59.669 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.130360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139596.log 2024-04-24T01:24:59.671 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31391.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.130360.log.gz 2024-04-24T01:24:59.671 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35266.log 2024-04-24T01:24:59.672 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139596.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139596.log.gz 2024-04-24T01:24:59.672 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56569.log 2024-04-24T01:24:59.673 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76047.log 2024-04-24T01:24:59.673 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88581.log 2024-04-24T01:24:59.675 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114233.log 2024-04-24T01:24:59.675 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76047.log: /var/log/ceph/cluster2-client.mirror.88581.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.57304.log.gz 2024-04-24T01:24:59.675 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98400.log 2024-04-24T01:24:59.675 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123611.log 2024-04-24T01:24:59.675 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114233.log: /var/log/ceph/cluster1-client.mirror.98400.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114233.log.gz 2024-04-24T01:24:59.675 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73386.log.gz 2024-04-24T01:24:59.676 INFO:teuthology.orchestra.run.smithi134.stderr: 81.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83786.log.gz 2024-04-24T01:24:59.677 INFO:teuthology.orchestra.run.smithi134.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98400.log.gz 2024-04-24T01:24:59.677 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95046.log 2024-04-24T01:24:59.677 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138818.log 2024-04-24T01:24:59.677 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84448.log.gz 2024-04-24T01:24:59.677 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95046.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94095.log.gz 2024-04-24T01:24:59.678 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68111.log 2024-04-24T01:24:59.678 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.92988.log.gz 2024-04-24T01:24:59.678 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138818.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138818.log.gz 2024-04-24T01:24:59.678 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123842.log 2024-04-24T01:24:59.679 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.68111.log: -- replaced with /var/log/ceph/cluster2-client.mirror.31391.log.gz 2024-04-24T01:24:59.680 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66462.log.gz 2024-04-24T01:24:59.681 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63211.log.gz 2024-04-24T01:24:59.682 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56569.log.gz 2024-04-24T01:24:59.682 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45965.log.gz 2024-04-24T01:24:59.682 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49678.log.gz 2024-04-24T01:24:59.695 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136004.log 2024-04-24T01:24:59.698 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123842.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70106.log.gz 2024-04-24T01:24:59.698 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83248.log.gz 2024-04-24T01:24:59.698 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29651.log.gz 2024-04-24T01:24:59.698 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123842.log.gz 2024-04-24T01:24:59.708 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58923.log 2024-04-24T01:24:59.708 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136004.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.136004.log.gz 2024-04-24T01:24:59.709 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.114009.log.gz 2024-04-24T01:24:59.710 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85344.log 2024-04-24T01:24:59.710 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137309.log 2024-04-24T01:24:59.713 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59623.log 2024-04-24T01:24:59.713 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137309.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137309.log.gz 2024-04-24T01:24:59.713 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82249.log 2024-04-24T01:24:59.713 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99729.log 2024-04-24T01:24:59.713 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70355.log 2024-04-24T01:24:59.724 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99729.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59623.log.gz 2024-04-24T01:24:59.724 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71855.log.gz 2024-04-24T01:24:59.725 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85344.log.gz 2024-04-24T01:24:59.727 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58923.log.gz 2024-04-24T01:24:59.727 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48770.log 2024-04-24T01:24:59.728 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70355.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68111.log.gz 2024-04-24T01:24:59.728 INFO:teuthology.orchestra.run.smithi134.stderr: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75899.log.gz 2024-04-24T01:24:59.729 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88581.log.gz 2024-04-24T01:24:59.730 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34429.log.gz 2024-04-24T01:24:59.730 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35266.log.gz 2024-04-24T01:24:59.731 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44545.log.gz 2024-04-24T01:24:59.731 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64466.log.gz 2024-04-24T01:24:59.731 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47123.log.gz 2024-04-24T01:24:59.731 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65901.log.gz 2024-04-24T01:24:59.733 INFO:teuthology.orchestra.run.smithi134.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.96344.log.gz 2024-04-24T01:24:59.744 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141882.log 2024-04-24T01:24:59.744 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82249.log.gz 2024-04-24T01:24:59.749 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48770.log: 93.6% 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81083.log.gz 2024-04-24T01:24:59.749 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.29724.log.gz 2024-04-24T01:24:59.750 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88966.log 2024-04-24T01:24:59.750 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75175.log.gz 2024-04-24T01:24:59.751 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141882.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141882.log.gz 2024-04-24T01:24:59.751 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48770.log.gz 2024-04-24T01:24:59.751 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79309.log 2024-04-24T01:24:59.751 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69371.log 2024-04-24T01:24:59.752 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98875.log 2024-04-24T01:24:59.753 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69371.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88966.log.gz 2024-04-24T01:24:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79309.log.gz 2024-04-24T01:24:59.755 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89530.log 2024-04-24T01:24:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69371.log.gz 2024-04-24T01:24:59.755 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98875.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98875.log.gz 2024-04-24T01:24:59.757 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47414.log 2024-04-24T01:24:59.757 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133548.log 2024-04-24T01:24:59.757 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47414.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37372.log.gz 2024-04-24T01:24:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.89530.log.gz 2024-04-24T01:24:59.758 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79729.log 2024-04-24T01:24:59.759 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133548.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133548.log.gz 2024-04-24T01:24:59.759 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82424.log 2024-04-24T01:24:59.759 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61444.log 2024-04-24T01:24:59.759 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82319.log 2024-04-24T01:24:59.760 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44219.log 2024-04-24T01:24:59.760 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128768.log 2024-04-24T01:24:59.761 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139661.log 2024-04-24T01:24:59.762 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128768.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128768.log.gz 2024-04-24T01:24:59.762 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41190.log 2024-04-24T01:24:59.763 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139661.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.79729.log.gz 2024-04-24T01:24:59.764 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.139661.log.gz 2024-04-24T01:24:59.764 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82319.log.gz 2024-04-24T01:24:59.765 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.95089.log.gz 2024-04-24T01:24:59.765 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82424.log.gz 2024-04-24T01:24:59.766 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57114.log 2024-04-24T01:24:59.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94800.log 2024-04-24T01:24:59.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.57114.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61444.log.gz 2024-04-24T01:24:59.769 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44219.log.gz 2024-04-24T01:24:59.769 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41190.log.gz 2024-04-24T01:24:59.770 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136567.log 2024-04-24T01:24:59.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94800.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39991.log 2024-04-24T01:24:59.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136567.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136567.log.gz 2024-04-24T01:24:59.770 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78675.log 2024-04-24T01:24:59.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39991.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86219.log.gz 2024-04-24T01:24:59.772 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54400.log 2024-04-24T01:24:59.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103677.log 2024-04-24T01:24:59.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.57182.log 2024-04-24T01:24:59.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.103677.log: 88.2% 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91283.log.gz 2024-04-24T01:24:59.775 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.103677.log.gz 2024-04-24T01:24:59.775 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129289.log 2024-04-24T01:24:59.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.57182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72021.log 2024-04-24T01:24:59.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129289.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94800.log.gz 2024-04-24T01:24:59.776 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78675.log.gz 2024-04-24T01:24:59.779 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39991.log.gz 2024-04-24T01:24:59.779 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104737.log.gz 2024-04-24T01:24:59.779 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54400.log.gz 2024-04-24T01:24:59.780 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129289.log.gz 2024-04-24T01:24:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129693.log 2024-04-24T01:24:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32286.log 2024-04-24T01:24:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137332.log 2024-04-24T01:24:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129693.log: /var/log/ceph/cluster2-client.mirror.32286.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.95046.log.gz 2024-04-24T01:24:59.780 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72021.log.gz 2024-04-24T01:24:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55758.log 2024-04-24T01:24:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137332.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137332.log.gz 2024-04-24T01:24:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131649.log 2024-04-24T01:24:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118242.log 2024-04-24T01:24:59.794 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131649.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123611.log.gz 2024-04-24T01:24:59.794 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32286.log.gz 2024-04-24T01:24:59.794 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92523.log 2024-04-24T01:24:59.795 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118242.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118242.log.gz 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131649.log.gz 2024-04-24T01:24:59.795 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:59.795 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114978.log 2024-04-24T01:24:59.796 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.92523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44739.log 2024-04-24T01:24:59.796 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114978.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135049.log 2024-04-24T01:24:59.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44739.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114978.log.gz 2024-04-24T01:24:59.797 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37275.log 2024-04-24T01:24:59.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.135049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89176.log 2024-04-24T01:24:59.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112798.log 2024-04-24T01:24:59.798 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135049.log.gz 2024-04-24T01:24:59.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89176.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55758.log.gz 2024-04-24T01:24:59.799 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37275.log.gz 2024-04-24T01:24:59.799 INFO:teuthology.orchestra.run.smithi134.stderr: 82.5% -- replaced with /var/log/ceph/cluster1-client.mirror.92523.log.gz 2024-04-24T01:24:59.799 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90688.log 2024-04-24T01:24:59.799 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.112798.log: -- replaced with /var/log/ceph/cluster2-client.mirror.44739.log.gz 2024-04-24T01:24:59.799 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.89176.log.gz 2024-04-24T01:24:59.800 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112798.log.gz 2024-04-24T01:24:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109695.log 2024-04-24T01:24:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46867.log 2024-04-24T01:24:59.801 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32565.log 2024-04-24T01:24:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109695.log: /var/log/ceph/cluster2-client.mirror.46867.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49581.log 2024-04-24T01:24:59.802 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.109695.log.gz 2024-04-24T01:24:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122776.log 2024-04-24T01:24:59.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49581.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90688.log.gz 2024-04-24T01:24:59.806 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140472.log 2024-04-24T01:24:59.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122776.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.122776.log.gz 2024-04-24T01:24:59.807 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29374.log 2024-04-24T01:24:59.808 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140472.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140472.log.gz 2024-04-24T01:24:59.809 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46867.log.gz 2024-04-24T01:24:59.809 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68566.log 2024-04-24T01:24:59.809 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49581.log.gz 2024-04-24T01:24:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29374.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29374.log.gz 2024-04-24T01:24:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133710.log 2024-04-24T01:24:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68566.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41935.log 2024-04-24T01:24:59.813 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133710.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133710.log.gz 2024-04-24T01:24:59.814 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38862.log 2024-04-24T01:24:59.814 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41935.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67481.log 2024-04-24T01:24:59.815 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105752.log 2024-04-24T01:24:59.816 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70635.log 2024-04-24T01:24:59.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105752.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32565.log.gz 2024-04-24T01:24:59.818 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48132.log 2024-04-24T01:24:59.819 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.46543.log 2024-04-24T01:24:59.820 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48132.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68566.log.gz 2024-04-24T01:24:59.820 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83958.log 2024-04-24T01:24:59.820 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.3.46543.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41935.log.gz 2024-04-24T01:24:59.820 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38862.log.gz 2024-04-24T01:24:59.832 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117879.log.gz 2024-04-24T01:24:59.834 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32007.log 2024-04-24T01:24:59.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83958.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111712.log 2024-04-24T01:24:59.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32007.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83958.log.gz 2024-04-24T01:24:59.840 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48132.log.gz 2024-04-24T01:24:59.840 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32007.log.gz 2024-04-24T01:24:59.846 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47414.log.gz 2024-04-24T01:24:59.854 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108371.log 2024-04-24T01:24:59.855 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111712.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111712.log.gz 2024-04-24T01:24:59.856 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125224.log 2024-04-24T01:24:59.857 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.108371.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108371.log.gz 2024-04-24T01:24:59.857 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137545.log 2024-04-24T01:24:59.858 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125224.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125224.log.gz 2024-04-24T01:24:59.858 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47352.log 2024-04-24T01:24:59.859 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.137545.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.137545.log.gz 2024-04-24T01:24:59.860 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.126773.log.gz 2024-04-24T01:24:59.868 INFO:teuthology.orchestra.run.smithi134.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99729.log.gz 2024-04-24T01:24:59.868 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129974.log 2024-04-24T01:24:59.869 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89918.log 2024-04-24T01:24:59.869 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56747.log 2024-04-24T01:24:59.869 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89918.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73141.log 2024-04-24T01:24:59.869 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.129974.log.gz 2024-04-24T01:24:59.871 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32600.log 2024-04-24T01:24:59.871 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43183.log 2024-04-24T01:24:59.871 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29244.log 2024-04-24T01:24:59.871 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46605.log 2024-04-24T01:24:59.871 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43183.log: /var/log/ceph/cluster2-client.admin.29244.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122117.log 2024-04-24T01:24:59.871 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.29244.log.gz 2024-04-24T01:24:59.872 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43478.log 2024-04-24T01:24:59.873 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122117.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73176.log 2024-04-24T01:24:59.873 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37504.log 2024-04-24T01:24:59.873 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.122117.log.gz 2024-04-24T01:24:59.873 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91598.log 2024-04-24T01:24:59.873 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73176.log: /var/log/ceph/cluster2-client.mirror.37504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125899.log 2024-04-24T01:24:59.873 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91598.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41741.log 2024-04-24T01:24:59.873 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95003.log 2024-04-24T01:24:59.875 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41741.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125899.log.gz 2024-04-24T01:24:59.875 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94867.log 2024-04-24T01:24:59.875 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95003.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47352.log.gz 2024-04-24T01:24:59.875 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73141.log.gz 2024-04-24T01:24:59.875 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73176.log.gz 2024-04-24T01:24:59.876 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32600.log.gz 2024-04-24T01:24:59.876 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43183.log.gz 2024-04-24T01:24:59.876 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42385.log 2024-04-24T01:24:59.876 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.94867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107845.log 2024-04-24T01:24:59.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.42385.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72406.log 2024-04-24T01:24:59.878 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.107845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35208.log 2024-04-24T01:24:59.878 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74335.log 2024-04-24T01:24:59.878 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.107845.log.gz 2024-04-24T01:24:59.878 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98796.log 2024-04-24T01:24:59.878 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35208.log: /var/log/ceph/cluster2-client.mirror.74335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95261.log 2024-04-24T01:24:59.878 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134341.log 2024-04-24T01:24:59.878 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95261.log: /var/log/ceph/cluster2-client.mirror.98796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136162.log 2024-04-24T01:24:59.879 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134341.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134341.log.gz 2024-04-24T01:24:59.883 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.97260.log.gz 2024-04-24T01:24:59.883 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129459.log 2024-04-24T01:24:59.885 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132617.log 2024-04-24T01:24:59.885 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136162.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136162.log.gz 2024-04-24T01:24:59.885 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113061.log 2024-04-24T01:24:59.885 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.132617.log: -- replaced with /var/log/ceph/cluster2-client.mirror.129459.log.gz 2024-04-24T01:24:59.886 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132617.log.gz 2024-04-24T01:24:59.887 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.81152.log 2024-04-24T01:24:59.888 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106117.log 2024-04-24T01:24:59.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.81152.log: /var/log/ceph/cluster2-client.mirror.113061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88896.log 2024-04-24T01:24:59.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.106117.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113061.log.gz 2024-04-24T01:24:59.890 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124735.log 2024-04-24T01:24:59.890 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106117.log.gz 2024-04-24T01:24:59.890 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-04-24T01:24:59.890 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88896.log: /var/log/ceph/cluster2-client.mirror.124735.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125628.log 2024-04-24T01:24:59.890 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124735.log.gz 2024-04-24T01:24:59.890 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127727.log 2024-04-24T01:24:59.890 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51812.log 2024-04-24T01:24:59.891 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125628.log.gz 2024-04-24T01:24:59.891 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137444.log 2024-04-24T01:24:59.893 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51812.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.127727.log.gz 2024-04-24T01:24:59.893 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39797.log 2024-04-24T01:24:59.894 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61129.log 2024-04-24T01:24:59.895 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137444.log: /var/log/ceph/cluster2-client.mirror.39797.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.42385.log.gz 2024-04-24T01:24:59.895 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137444.log.gz 2024-04-24T01:24:59.895 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128157.log 2024-04-24T01:24:59.895 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43637.log 2024-04-24T01:24:59.897 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128157.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65496.log 2024-04-24T01:24:59.897 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43637.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128157.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30004.log 2024-04-24T01:24:59.897 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:24:59.897 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133497.log 2024-04-24T01:24:59.897 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65496.log: /var/log/ceph/cluster1-client.admin.30004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73911.log 2024-04-24T01:24:59.897 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36287.log 2024-04-24T01:24:59.898 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81724.log 2024-04-24T01:24:59.898 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98243.log 2024-04-24T01:24:59.899 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81724.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133497.log.gz 2024-04-24T01:24:59.899 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45577.log 2024-04-24T01:24:59.899 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110262.log 2024-04-24T01:24:59.899 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98243.log: /var/log/ceph/cluster2-client.mirror.45577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65181.log 2024-04-24T01:24:59.900 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94622.log 2024-04-24T01:24:59.902 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65181.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37504.log.gz 2024-04-24T01:24:59.904 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110262.log: gzip 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110262.log.gz -5 2024-04-24T01:24:59.904 INFO:teuthology.orchestra.run.smithi134.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.60289.log 2024-04-24T01:24:59.904 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72406.log.gz 2024-04-24T01:24:59.904 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130905.log 2024-04-24T01:24:59.905 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53624.log 2024-04-24T01:24:59.905 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31090.log 2024-04-24T01:24:59.905 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53624.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130905.log.gz 2024-04-24T01:24:59.906 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138457.log 2024-04-24T01:24:59.906 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.31090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75814.log 2024-04-24T01:24:59.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108304.log 2024-04-24T01:24:59.908 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.75814.log: -- replaced with /var/log/ceph/cluster1-client.mirror.138457.log.gz 2024-04-24T01:24:59.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41776.log 2024-04-24T01:24:59.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113477.log 2024-04-24T01:24:59.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41776.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108304.log.gz 2024-04-24T01:24:59.911 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104261.log 2024-04-24T01:24:59.911 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78465.log 2024-04-24T01:24:59.911 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113477.log.gz 2024-04-24T01:24:59.912 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117216.log 2024-04-24T01:24:59.912 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78465.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99646.log 2024-04-24T01:24:59.913 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117216.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117216.log.gz 2024-04-24T01:24:59.913 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94388.log 2024-04-24T01:24:59.925 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99646.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89918.log.gz 2024-04-24T01:24:59.927 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-04-24T01:24:59.927 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94388.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.94622.log.gz 2024-04-24T01:24:59.931 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91598.log.gz 2024-04-24T01:24:59.933 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94867.log.gz 2024-04-24T01:24:59.935 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56747.log.gz 2024-04-24T01:24:59.938 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.100968.log 2024-04-24T01:24:59.942 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.81152.log.gz 2024-04-24T01:24:59.945 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94388.log.gz 2024-04-24T01:24:59.945 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65181.log.gz 2024-04-24T01:24:59.951 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.88896.log.gz 2024-04-24T01:24:59.952 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.3.100968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.99526.log 2024-04-24T01:24:59.959 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46605.log.gz 2024-04-24T01:24:59.960 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster2.log.gz 2024-04-24T01:24:59.962 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60289.log.gz 2024-04-24T01:24:59.963 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61129.log.gz 2024-04-24T01:24:59.964 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73911.log.gz 2024-04-24T01:24:59.964 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74335.log.gz 2024-04-24T01:24:59.966 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43478.log.gz 2024-04-24T01:24:59.966 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100693.log 2024-04-24T01:24:59.968 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.99526.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94983.log.gz 2024-04-24T01:24:59.982 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76120.log 2024-04-24T01:24:59.990 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.100693.log: 94.6% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36287.log.gz 2024-04-24T01:24:59.993 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51812.log.gz 2024-04-24T01:24:59.993 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.100693.log.gz 2024-04-24T01:25:00.003 INFO:teuthology.orchestra.run.smithi134.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.30004.log.gz 2024-04-24T01:25:00.003 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80534.log 2024-04-24T01:25:00.010 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76120.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81724.log.gz 2024-04-24T01:25:00.018 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76120.log.gz 2024-04-24T01:25:00.019 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46388.log 2024-04-24T01:25:00.019 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41776.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.41741.log.gz 2024-04-24T01:25:00.019 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:00.024 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80534.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129693.log.gz 2024-04-24T01:25:00.025 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52262.log 2024-04-24T01:25:00.025 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140847.log 2024-04-24T01:25:00.030 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52262.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45577.log.gz 2024-04-24T01:25:00.031 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46388.log.gz 2024-04-24T01:25:00.032 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52262.log.gz 2024-04-24T01:25:00.041 INFO:teuthology.orchestra.run.smithi134.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.99526.log.gz 2024-04-24T01:25:00.041 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99215.log 2024-04-24T01:25:00.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140847.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140847.log.gz 2024-04-24T01:25:00.043 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95803.log 2024-04-24T01:25:00.050 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99215.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70355.log.gz 2024-04-24T01:25:00.050 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70635.log.gz 2024-04-24T01:25:00.050 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29749.log 2024-04-24T01:25:00.050 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95803.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78465.log.gz 2024-04-24T01:25:00.055 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35208.log.gz 2024-04-24T01:25:00.056 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39797.log.gz 2024-04-24T01:25:00.061 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53624.log.gz 2024-04-24T01:25:00.063 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43637.log.gz 2024-04-24T01:25:00.070 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65496.log.gz 2024-04-24T01:25:00.070 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73071.log 2024-04-24T01:25:00.086 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.29749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135869.log 2024-04-24T01:25:00.089 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73071.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.31090.log.gz 2024-04-24T01:25:00.089 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73071.log.gz 2024-04-24T01:25:00.095 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135869.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135869.log.gz 2024-04-24T01:25:00.096 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56049.log 2024-04-24T01:25:00.109 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95803.log.gz 2024-04-24T01:25:00.111 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97790.log.gz 2024-04-24T01:25:00.112 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134065.log 2024-04-24T01:25:00.112 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34185.log 2024-04-24T01:25:00.115 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134065.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134065.log.gz 2024-04-24T01:25:00.115 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126103.log 2024-04-24T01:25:00.116 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34185.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.98243.log.gz 2024-04-24T01:25:00.116 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80254.log 2024-04-24T01:25:00.117 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.126103.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126103.log.gz 2024-04-24T01:25:00.117 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97561.log 2024-04-24T01:25:00.118 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30157.log 2024-04-24T01:25:00.118 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73876.log 2024-04-24T01:25:00.121 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97561.log: /var/log/ceph/cluster2-client.admin.30157.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30157.log.gz 2024-04-24T01:25:00.124 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34615.log 2024-04-24T01:25:00.124 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63596.log 2024-04-24T01:25:00.125 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86881.log 2024-04-24T01:25:00.126 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90128.log 2024-04-24T01:25:00.127 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7%/var/log/ceph/cluster1-client.mirror.86881.log: -- replaced with /var/log/ceph/cluster2-client.mirror.97561.log.gz 2024-04-24T01:25:00.129 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80534.log.gz 2024-04-24T01:25:00.131 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34185.log.gz 2024-04-24T01:25:00.134 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34615.log.gz 2024-04-24T01:25:00.134 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73876.log.gz 2024-04-24T01:25:00.135 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63596.log.gz 2024-04-24T01:25:00.137 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105752.log.gz 2024-04-24T01:25:00.138 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79624.log 2024-04-24T01:25:00.139 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90128.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109501.log 2024-04-24T01:25:00.139 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107999.log 2024-04-24T01:25:00.139 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95261.log.gz 2024-04-24T01:25:00.140 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109501.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.109501.log.gz 2024-04-24T01:25:00.141 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71195.log 2024-04-24T01:25:00.142 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107999.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107999.log.gz 2024-04-24T01:25:00.142 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91528.log 2024-04-24T01:25:00.143 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71195.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86881.log.gz 2024-04-24T01:25:00.143 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32228.log 2024-04-24T01:25:00.143 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129138.log 2024-04-24T01:25:00.144 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122159.log 2024-04-24T01:25:00.144 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129138.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129138.log.gz 2024-04-24T01:25:00.145 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44607.log 2024-04-24T01:25:00.146 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122159.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.122159.log.gz 2024-04-24T01:25:00.146 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109990.log 2024-04-24T01:25:00.146 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44607.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.90128.log.gz 2024-04-24T01:25:00.148 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-04-24T01:25:00.150 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32228.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56049.log.gz 2024-04-24T01:25:00.152 INFO:teuthology.orchestra.run.smithi134.stderr: 85.7% -- replaced with /var/log/ceph/cluster1-client.mirror.29749.log.gz 2024-04-24T01:25:00.152 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79624.log.gz 2024-04-24T01:25:00.152 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71195.log.gz 2024-04-24T01:25:00.152 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44607.log.gz 2024-04-24T01:25:00.159 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40899.log 2024-04-24T01:25:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109990.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66956.log 2024-04-24T01:25:00.160 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109990.log.gz 2024-04-24T01:25:00.161 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114655.log 2024-04-24T01:25:00.161 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88371.log 2024-04-24T01:25:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114655.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114655.log.gz 2024-04-24T01:25:00.164 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50322.log 2024-04-24T01:25:00.164 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103310.log 2024-04-24T01:25:00.165 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43249.log 2024-04-24T01:25:00.166 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50322.log: /var/log/ceph/cluster2-client.mirror.103310.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103310.log.gz 2024-04-24T01:25:00.166 INFO:teuthology.orchestra.run.smithi134.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98796.log.gz 2024-04-24T01:25:00.166 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133242.log 2024-04-24T01:25:00.166 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40899.log.gz 2024-04-24T01:25:00.168 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43249.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50322.log.gz 2024-04-24T01:25:00.169 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66956.log.gz 2024-04-24T01:25:00.169 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.95003.log.gz 2024-04-24T01:25:00.169 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54788.log 2024-04-24T01:25:00.170 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96307.log 2024-04-24T01:25:00.170 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28805.log 2024-04-24T01:25:00.171 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133242.log.gz 2024-04-24T01:25:00.171 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41675.log 2024-04-24T01:25:00.171 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.28805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61514.log 2024-04-24T01:25:00.172 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28805.log.gz 2024-04-24T01:25:00.172 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44801.log 2024-04-24T01:25:00.172 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29524.log 2024-04-24T01:25:00.173 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88791.log 2024-04-24T01:25:00.174 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125663.log 2024-04-24T01:25:00.174 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91528.log.gz 2024-04-24T01:25:00.175 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29524.log: /var/log/ceph/cluster2-client.mirror.88791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49934.log 2024-04-24T01:25:00.175 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29524.log.gz 2024-04-24T01:25:00.175 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60499.log 2024-04-24T01:25:00.175 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125663.log.gz 2024-04-24T01:25:00.175 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133739.log 2024-04-24T01:25:00.175 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49934.log: /var/log/ceph/cluster2-client.mirror.60499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119575.log 2024-04-24T01:25:00.176 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133739.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.133739.log.gz 2024-04-24T01:25:00.176 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88371.log.gz 2024-04-24T01:25:00.176 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42857.log 2024-04-24T01:25:00.177 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80763.log 2024-04-24T01:25:00.177 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43249.log.gz 2024-04-24T01:25:00.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119575.log: /var/log/ceph/cluster2-client.mirror.42857.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84133.log 2024-04-24T01:25:00.182 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119575.log.gz 2024-04-24T01:25:00.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111528.log 2024-04-24T01:25:00.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84133.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96055.log 2024-04-24T01:25:00.182 INFO:teuthology.orchestra.run.smithi134.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99215.log.gz 2024-04-24T01:25:00.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113442.log 2024-04-24T01:25:00.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133072.log 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37698.log 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113442.log: /var/log/ceph/cluster2-client.mirror.133072.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54788.log.gz 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113442.log.gz 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56084.log 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133072.log.gz 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37698.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72861.log 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61514.log.gz 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56278.log 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132436.log 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72861.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133355.log 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56278.log: /var/log/ceph/cluster2-client.mirror.132436.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48964.log 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133355.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.132436.log.gz 2024-04-24T01:25:00.185 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88791.log.gz 2024-04-24T01:25:00.186 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41675.log.gz 2024-04-24T01:25:00.186 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133355.log.gz 2024-04-24T01:25:00.186 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44801.log.gz 2024-04-24T01:25:00.186 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60499.log.gz 2024-04-24T01:25:00.186 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116082.log 2024-04-24T01:25:00.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46000.log 2024-04-24T01:25:00.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116082.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86114.log 2024-04-24T01:25:00.188 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46000.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72861.log.gz 2024-04-24T01:25:00.188 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84133.log.gz 2024-04-24T01:25:00.189 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42857.log.gz 2024-04-24T01:25:00.189 INFO:teuthology.orchestra.run.smithi134.stderr: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.111528.log.gz 2024-04-24T01:25:00.190 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101973.log 2024-04-24T01:25:00.190 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84483.log 2024-04-24T01:25:00.190 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116082.log.gz 2024-04-24T01:25:00.191 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113349.log 2024-04-24T01:25:00.192 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141071.log 2024-04-24T01:25:00.192 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36702.log 2024-04-24T01:25:00.192 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0%/var/log/ceph/cluster1-client.mirror.141071.log: -- replaced with /var/log/ceph/cluster2-client.mirror.113349.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41640.log 2024-04-24T01:25:00.192 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:00.193 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141071.log.gz 2024-04-24T01:25:00.193 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.36702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125162.log 2024-04-24T01:25:00.193 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49934.log.gz 2024-04-24T01:25:00.193 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41640.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96055.log.gz 2024-04-24T01:25:00.194 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48964.log.gz 2024-04-24T01:25:00.194 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36194.log 2024-04-24T01:25:00.194 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86906.log 2024-04-24T01:25:00.195 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125162.log.gz 2024-04-24T01:25:00.195 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140224.log 2024-04-24T01:25:00.195 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115503.log 2024-04-24T01:25:00.196 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140224.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56084.log.gz 2024-04-24T01:25:00.196 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56278.log.gz 2024-04-24T01:25:00.196 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.140224.log.gz 2024-04-24T01:25:00.196 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46000.log.gz 2024-04-24T01:25:00.197 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86114.log.gz 2024-04-24T01:25:00.197 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136364.log 2024-04-24T01:25:00.198 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115503.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37698.log.gz 2024-04-24T01:25:00.198 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.71784.log 2024-04-24T01:25:00.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121729.log 2024-04-24T01:25:00.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136364.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136364.log.gz 2024-04-24T01:25:00.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.71784.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84483.log.gz 2024-04-24T01:25:00.199 INFO:teuthology.orchestra.run.smithi134.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115503.log.gz 2024-04-24T01:25:00.200 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43148.log 2024-04-24T01:25:00.201 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103610.log 2024-04-24T01:25:00.201 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43148.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80941.log 2024-04-24T01:25:00.201 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121729.log.gz 2024-04-24T01:25:00.201 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103610.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103610.log.gz 2024-04-24T01:25:00.203 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41093.log 2024-04-24T01:25:00.203 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72546.log 2024-04-24T01:25:00.203 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108978.log 2024-04-24T01:25:00.203 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140782.log 2024-04-24T01:25:00.203 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119041.log 2024-04-24T01:25:00.205 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108978.log: /var/log/ceph/cluster1-client.mirror.140782.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108978.log.gz 2024-04-24T01:25:00.205 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95632.log 2024-04-24T01:25:00.205 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.140782.log.gz 2024-04-24T01:25:00.205 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36194.log.gz 2024-04-24T01:25:00.205 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115013.log 2024-04-24T01:25:00.205 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119041.log.gz 2024-04-24T01:25:00.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95632.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121947.log 2024-04-24T01:25:00.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57488.log 2024-04-24T01:25:00.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121947.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115013.log.gz 2024-04-24T01:25:00.206 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33871.log 2024-04-24T01:25:00.207 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121947.log.gz 2024-04-24T01:25:00.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30183.log 2024-04-24T01:25:00.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127436.log 2024-04-24T01:25:00.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.30183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139721.log 2024-04-24T01:25:00.208 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.127436.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67621.log 2024-04-24T01:25:00.208 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139721.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127436.log.gz 2024-04-24T01:25:00.209 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42504.log 2024-04-24T01:25:00.210 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.139721.log.gz 2024-04-24T01:25:00.210 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67621.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73806.log 2024-04-24T01:25:00.210 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115446.log 2024-04-24T01:25:00.210 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42504.log: /var/log/ceph/cluster2-client.mirror.73806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141041.log 2024-04-24T01:25:00.210 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115446.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115446.log.gz 2024-04-24T01:25:00.212 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117445.log 2024-04-24T01:25:00.212 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36831.log 2024-04-24T01:25:00.212 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117445.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117445.log.gz 2024-04-24T01:25:00.214 INFO:teuthology.orchestra.run.smithi134.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141041.log.gz 2024-04-24T01:25:00.216 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49740.log 2024-04-24T01:25:00.216 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136837.log 2024-04-24T01:25:00.217 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109122.log 2024-04-24T01:25:00.217 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110127.log 2024-04-24T01:25:00.218 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136837.log.gz 2024-04-24T01:25:00.218 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34650.log 2024-04-24T01:25:00.221 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110127.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109122.log.gz 2024-04-24T01:25:00.222 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141778.log 2024-04-24T01:25:00.222 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110127.log.gz 2024-04-24T01:25:00.222 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54982.log 2024-04-24T01:25:00.223 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34650.log: /var/log/ceph/cluster2-client.admin.141778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141778.log.gz 2024-04-24T01:25:00.226 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93793.log 2024-04-24T01:25:00.227 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54982.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43866.log 2024-04-24T01:25:00.227 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97008.log 2024-04-24T01:25:00.228 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82669.log 2024-04-24T01:25:00.229 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75722.log 2024-04-24T01:25:00.230 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76330.log 2024-04-24T01:25:00.230 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76155.log 2024-04-24T01:25:00.230 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95695.log 2024-04-24T01:25:00.230 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122687.log 2024-04-24T01:25:00.231 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76155.log: /var/log/ceph/cluster1-client.mirror.95695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123958.log 2024-04-24T01:25:00.231 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122687.log: /var/log/ceph/cluster2-client.mirror.123958.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122687.log.gz 2024-04-24T01:25:00.231 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66112.log 2024-04-24T01:25:00.231 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123958.log.gz 2024-04-24T01:25:00.232 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118516.log 2024-04-24T01:25:00.232 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121120.log 2024-04-24T01:25:00.232 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66112.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75857.log 2024-04-24T01:25:00.234 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.118516.log: /var/log/ceph/cluster2-client.mirror.121120.log: 91.9% 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118516.log.gz 2024-04-24T01:25:00.234 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.121120.log.gz 2024-04-24T01:25:00.235 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101443.log 2024-04-24T01:25:00.235 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75857.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95782.log 2024-04-24T01:25:00.236 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101443.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101443.log.gz 2024-04-24T01:25:00.236 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42989.log 2024-04-24T01:25:00.236 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76225.log 2024-04-24T01:25:00.236 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95782.log: /var/log/ceph/cluster2-client.mirror.42989.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86324.log 2024-04-24T01:25:00.238 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74195.log 2024-04-24T01:25:00.238 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64766.log 2024-04-24T01:25:00.238 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102784.log 2024-04-24T01:25:00.238 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.64766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112547.log 2024-04-24T01:25:00.239 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.102784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113967.log 2024-04-24T01:25:00.239 INFO:teuthology.orchestra.run.smithi134.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.102784.log.gz 2024-04-24T01:25:00.240 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.112547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54594.log 2024-04-24T01:25:00.240 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113967.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112547.log.gz 2024-04-24T01:25:00.240 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133313.log 2024-04-24T01:25:00.240 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54594.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82669.log.gz 2024-04-24T01:25:00.240 INFO:teuthology.orchestra.run.smithi134.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.113967.log.gz 2024-04-24T01:25:00.241 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85904.log 2024-04-24T01:25:00.241 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46159.log 2024-04-24T01:25:00.242 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85904.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133313.log.gz 2024-04-24T01:25:00.242 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.62970.log 2024-04-24T01:25:00.243 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105365.log 2024-04-24T01:25:00.243 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.62970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134878.log 2024-04-24T01:25:00.244 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121912.log 2024-04-24T01:25:00.244 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105365.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105365.log.gz 2024-04-24T01:25:00.244 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39312.log 2024-04-24T01:25:00.245 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121912.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121912.log.gz 2024-04-24T01:25:00.245 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109752.log 2024-04-24T01:25:00.246 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116005.log 2024-04-24T01:25:00.247 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109752.log: /var/log/ceph/cluster2-client.mirror.134878.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109752.log.gz 2024-04-24T01:25:00.248 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136747.log 2024-04-24T01:25:00.248 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80254.log.gz 2024-04-24T01:25:00.249 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116005.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134878.log.gz 2024-04-24T01:25:00.249 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116005.log.gz 2024-04-24T01:25:00.249 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95284.log 2024-04-24T01:25:00.250 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62449.log 2024-04-24T01:25:00.251 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124956.log 2024-04-24T01:25:00.251 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76225.log.gz 2024-04-24T01:25:00.251 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76330.log.gz 2024-04-24T01:25:00.251 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76155.log.gz 2024-04-24T01:25:00.251 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62449.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136747.log.gz 2024-04-24T01:25:00.251 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109606.log 2024-04-24T01:25:00.251 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.124956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138686.log 2024-04-24T01:25:00.252 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109606.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124956.log.gz 2024-04-24T01:25:00.253 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.109606.log.gz 2024-04-24T01:25:00.253 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39929.log 2024-04-24T01:25:00.253 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115734.log 2024-04-24T01:25:00.253 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39929.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.138686.log.gz 2024-04-24T01:25:00.253 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55502.log 2024-04-24T01:25:00.253 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115734.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115734.log.gz 2024-04-24T01:25:00.253 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123146.log 2024-04-24T01:25:00.254 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124168.log 2024-04-24T01:25:00.255 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123146.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.123146.log.gz 2024-04-24T01:25:00.255 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136657.log 2024-04-24T01:25:00.256 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95971.log 2024-04-24T01:25:00.256 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136657.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124168.log.gz 2024-04-24T01:25:00.257 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136657.log.gz 2024-04-24T01:25:00.257 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75070.log 2024-04-24T01:25:00.257 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66042.log 2024-04-24T01:25:00.257 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120918.log 2024-04-24T01:25:00.261 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113384.log 2024-04-24T01:25:00.261 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141615.log 2024-04-24T01:25:00.261 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.120918.log.gz 2024-04-24T01:25:00.261 INFO:teuthology.orchestra.run.smithi134.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82109.log 2024-04-24T01:25:00.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113384.log: /var/log/ceph/cluster1-client.admin.141615.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.141615.log.gz 2024-04-24T01:25:00.262 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88756.log 2024-04-24T01:25:00.262 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.113384.log.gz 2024-04-24T01:25:00.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34522.log 2024-04-24T01:25:00.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43769.log 2024-04-24T01:25:00.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95347.log 2024-04-24T01:25:00.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79799.log 2024-04-24T01:25:00.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132788.log 2024-04-24T01:25:00.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67726.log 2024-04-24T01:25:00.263 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51521.log 2024-04-24T01:25:00.265 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67726.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41640.log.gz 2024-04-24T01:25:00.265 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132788.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66112.log.gz 2024-04-24T01:25:00.265 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66042.log.gz 2024-04-24T01:25:00.265 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104261.log.gz 2024-04-24T01:25:00.265 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132788.log.gz 2024-04-24T01:25:00.265 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33813.log 2024-04-24T01:25:00.265 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51521.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73806.log.gz 2024-04-24T01:25:00.266 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72336.log 2024-04-24T01:25:00.266 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108416.log 2024-04-24T01:25:00.266 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.35909.log 2024-04-24T01:25:00.266 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72336.log: /var/log/ceph/cluster1-client.mirror.108416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99192.log 2024-04-24T01:25:00.266 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.35909.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80941.log.gz 2024-04-24T01:25:00.266 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42989.log.gz 2024-04-24T01:25:00.266 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43148.log.gz 2024-04-24T01:25:00.267 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57488.log.gz 2024-04-24T01:25:00.267 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62449.log.gz 2024-04-24T01:25:00.267 INFO:teuthology.orchestra.run.smithi134.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108416.log.gz 2024-04-24T01:25:00.269 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138562.log 2024-04-24T01:25:00.269 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50454.log 2024-04-24T01:25:00.269 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99192.log.gz 2024-04-24T01:25:00.269 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140727.log 2024-04-24T01:25:00.269 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138562.log.gz 2024-04-24T01:25:00.269 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57182.log 2024-04-24T01:25:00.269 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50454.log: /var/log/ceph/cluster1-client.mirror.140727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29499.log 2024-04-24T01:25:00.270 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.57182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123343.log 2024-04-24T01:25:00.270 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.140727.log.gz 2024-04-24T01:25:00.270 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.29499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29499.log.gz 2024-04-24T01:25:00.271 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.123343.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.123343.log.gz 2024-04-24T01:25:00.272 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138211.log 2024-04-24T01:25:00.272 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101950.log 2024-04-24T01:25:00.273 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44642.log 2024-04-24T01:25:00.273 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58783.log 2024-04-24T01:25:00.273 INFO:teuthology.orchestra.run.smithi134.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101950.log.gz 2024-04-24T01:25:00.273 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44642.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.138211.log.gz 2024-04-24T01:25:00.274 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104622.log 2024-04-24T01:25:00.274 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69161.log 2024-04-24T01:25:00.275 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.104622.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104622.log.gz 2024-04-24T01:25:00.277 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131933.log 2024-04-24T01:25:00.277 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74545.log 2024-04-24T01:25:00.278 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131933.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126637.log 2024-04-24T01:25:00.278 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62910.log 2024-04-24T01:25:00.279 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126637.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.46543.log.gz 2024-04-24T01:25:00.279 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126637.log.gz 2024-04-24T01:25:00.280 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95132.log 2024-04-24T01:25:00.280 INFO:teuthology.orchestra.run.smithi134.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131933.log.gz 2024-04-24T01:25:00.280 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80852.log 2024-04-24T01:25:00.280 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87951.log 2024-04-24T01:25:00.281 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42504.log.gz 2024-04-24T01:25:00.281 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80852.log: /var/log/ceph/cluster1-client.mirror.95132.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51521.log.gz 2024-04-24T01:25:00.282 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95519.log 2024-04-24T01:25:00.282 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54982.log.gz 2024-04-24T01:25:00.282 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89810.log 2024-04-24T01:25:00.282 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130315.log 2024-04-24T01:25:00.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121147.log 2024-04-24T01:25:00.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.130315.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99480.log 2024-04-24T01:25:00.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.121147.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96523.log 2024-04-24T01:25:00.284 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43866.log.gz 2024-04-24T01:25:00.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45383.log 2024-04-24T01:25:00.285 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130315.log.gz 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99480.log.gz 2024-04-24T01:25:00.285 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:00.285 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49740.log.gz 2024-04-24T01:25:00.285 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96523.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54594.log.gz 2024-04-24T01:25:00.285 INFO:teuthology.orchestra.run.smithi134.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121147.log.gz 2024-04-24T01:25:00.291 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133001.log 2024-04-24T01:25:00.291 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120686.log 2024-04-24T01:25:00.291 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72756.log 2024-04-24T01:25:00.291 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115912.log 2024-04-24T01:25:00.291 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96034.log 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133001.log.gz 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.120686.log.gz 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89740.log 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115912.log: /var/log/ceph/cluster2-client.mirror.96034.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77730.log 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115912.log.gz 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133639.log 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82774.log 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114521.log 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42129.log 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114521.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66637.log 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109299.log 2024-04-24T01:25:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42129.log: /var/log/ceph/cluster2-client.mirror.66637.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133639.log.gz 2024-04-24T01:25:00.296 INFO:teuthology.orchestra.run.smithi134.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114521.log.gz 2024-04-24T01:25:00.296 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52883.log 2024-04-24T01:25:00.296 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126432.log 2024-04-24T01:25:00.296 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109299.log.gz 2024-04-24T01:25:00.296 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121542.log 2024-04-24T01:25:00.296 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60604.log 2024-04-24T01:25:00.296 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.121542.log: -- replaced with /var/log/ceph/cluster2-client.mirror.126432.log.gz 2024-04-24T01:25:00.296 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91493.log 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60604.log: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36772.log 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.121542.log.gz 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135532.log 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.36772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138664.log 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45224.log 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.135532.log.gz 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95413.log 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131820.log 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.138664.log.gz 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86499.log 2024-04-24T01:25:00.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130283.log 2024-04-24T01:25:00.301 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134858.log 2024-04-24T01:25:00.301 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130283.log: 89.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134532.log 2024-04-24T01:25:00.301 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.131820.log.gz 2024-04-24T01:25:00.301 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130283.log.gz 2024-04-24T01:25:00.301 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30934.log 2024-04-24T01:25:00.301 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79904.log 2024-04-24T01:25:00.301 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.30934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139213.log 2024-04-24T01:25:00.301 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134858.log.gz 2024-04-24T01:25:00.301 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123298.log 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79904.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134532.log.gz 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115947.log 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.139213.log.gz 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.123298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138712.log 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70425.log 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123298.log.gz 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138712.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.115947.log.gz 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57026.log 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70425.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38051.log 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.57026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123425.log 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60079.log 2024-04-24T01:25:00.302 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123425.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87426.log 2024-04-24T01:25:00.303 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127657.log 2024-04-24T01:25:00.303 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87426.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123425.log.gz 2024-04-24T01:25:00.303 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72336.log.gz 2024-04-24T01:25:00.303 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75070.log.gz 2024-04-24T01:25:00.307 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72441.log 2024-04-24T01:25:00.307 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36537.log 2024-04-24T01:25:00.308 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.72441.log: -- replaced with /var/log/ceph/cluster2-client.mirror.127657.log.gz 2024-04-24T01:25:00.308 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45418.log 2024-04-24T01:25:00.308 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104797.log 2024-04-24T01:25:00.308 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36537.log: /var/log/ceph/cluster2-client.mirror.45418.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72931.log 2024-04-24T01:25:00.309 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95992.log 2024-04-24T01:25:00.309 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104797.log.gz 2024-04-24T01:25:00.309 INFO:teuthology.orchestra.run.smithi134.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95782.log.gz 2024-04-24T01:25:00.310 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98039.log 2024-04-24T01:25:00.311 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111615.log 2024-04-24T01:25:00.311 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68356.log 2024-04-24T01:25:00.311 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98039.log.gz 2024-04-24T01:25:00.311 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111615.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111615.log.gz 2024-04-24T01:25:00.312 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58167.log 2024-04-24T01:25:00.312 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42601.log 2024-04-24T01:25:00.313 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76960.log 2024-04-24T01:25:00.313 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70320.log 2024-04-24T01:25:00.314 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103399.log 2024-04-24T01:25:00.314 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95824.log 2024-04-24T01:25:00.314 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103839.log 2024-04-24T01:25:00.314 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.95824.log: -- replaced with /var/log/ceph/cluster2-client.mirror.103399.log.gz 2024-04-24T01:25:00.315 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94477.log 2024-04-24T01:25:00.315 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135487.log 2024-04-24T01:25:00.315 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94477.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103839.log.gz 2024-04-24T01:25:00.315 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52394.log 2024-04-24T01:25:00.316 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135487.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68776.log 2024-04-24T01:25:00.316 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.135487.log.gz 2024-04-24T01:25:00.316 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38280.log 2024-04-24T01:25:00.316 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68776.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100578.log 2024-04-24T01:25:00.317 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121461.log 2024-04-24T01:25:00.317 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100578.log: /var/log/ceph/cluster2-client.mirror.121461.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72546.log.gz 2024-04-24T01:25:00.317 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103702.log 2024-04-24T01:25:00.318 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121461.log.gz 2024-04-24T01:25:00.318 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74860.log 2024-04-24T01:25:00.319 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31670.log 2024-04-24T01:25:00.321 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103702.log: /var/log/ceph/cluster2-client.mirror.74860.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67481.log.gz 2024-04-24T01:25:00.321 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67621.log.gz 2024-04-24T01:25:00.321 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99646.log.gz 2024-04-24T01:25:00.321 INFO:teuthology.orchestra.run.smithi134.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.103702.log.gz 2024-04-24T01:25:00.321 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140826.log 2024-04-24T01:25:00.321 INFO:teuthology.orchestra.run.smithi134.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.100968.log.gz 2024-04-24T01:25:00.324 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59939.log 2024-04-24T01:25:00.324 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93793.log.gz 2024-04-24T01:25:00.324 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100668.log 2024-04-24T01:25:00.324 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133668.log 2024-04-24T01:25:00.325 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140826.log.gz/var/log/ceph/cluster2-client.mirror.59939.log: 2024-04-24T01:25:00.325 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34708.log 2024-04-24T01:25:00.325 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129546.log 2024-04-24T01:25:00.325 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126736.log 2024-04-24T01:25:00.325 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44025.log 2024-04-24T01:25:00.325 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133668.log.gz 2024-04-24T01:25:00.325 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129546.log: /var/log/ceph/cluster2-client.mirror.126736.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129546.log.gz 2024-04-24T01:25:00.326 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64331.log 2024-04-24T01:25:00.326 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127000.log 2024-04-24T01:25:00.326 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44025.log: /var/log/ceph/cluster2-client.mirror.64331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49193.log 2024-04-24T01:25:00.326 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135329.log 2024-04-24T01:25:00.327 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127000.log: /var/log/ceph/cluster2-client.mirror.49193.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.127000.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56712.log 2024-04-24T01:25:00.328 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:00.328 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45286.log 2024-04-24T01:25:00.328 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135329.log.gz 2024-04-24T01:25:00.328 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37857.log 2024-04-24T01:25:00.328 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56712.log: /var/log/ceph/cluster2-client.mirror.45286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59413.log 2024-04-24T01:25:00.329 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106618.log 2024-04-24T01:25:00.330 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37857.log: /var/log/ceph/cluster2-client.mirror.59413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117410.log 2024-04-24T01:25:00.331 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103364.log 2024-04-24T01:25:00.331 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106618.log.gz 2024-04-24T01:25:00.333 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114541.log 2024-04-24T01:25:00.333 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117410.log.gz 2024-04-24T01:25:00.333 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103364.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124087.log 2024-04-24T01:25:00.333 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.103364.log.gz 2024-04-24T01:25:00.333 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.114541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108835.log 2024-04-24T01:25:00.335 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124087.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124087.log.gz 2024-04-24T01:25:00.335 INFO:teuthology.orchestra.run.smithi134.stderr: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126736.log.gz 2024-04-24T01:25:00.335 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87741.log 2024-04-24T01:25:00.335 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105899.log 2024-04-24T01:25:00.335 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108835.log.gz 2024-04-24T01:25:00.335 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87741.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114541.log.gz 2024-04-24T01:25:00.336 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31068.log 2024-04-24T01:25:00.336 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57585.log 2024-04-24T01:25:00.336 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31068.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105899.log.gz 2024-04-24T01:25:00.337 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128369.log 2024-04-24T01:25:00.337 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42264.log 2024-04-24T01:25:00.338 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131032.log 2024-04-24T01:25:00.339 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136702.log 2024-04-24T01:25:00.339 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42892.log 2024-04-24T01:25:00.340 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131032.log: /var/log/ceph/cluster1-client.mirror.136702.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136702.log.gz 2024-04-24T01:25:00.340 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131032.log.gz 2024-04-24T01:25:00.340 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109571.log 2024-04-24T01:25:00.341 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96236.log 2024-04-24T01:25:00.344 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104715.log 2024-04-24T01:25:00.344 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109571.log.gz 2024-04-24T01:25:00.345 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.111259.log 2024-04-24T01:25:00.345 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.96236.log: /var/log/ceph/cluster1-client.mirror.104715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81899.log 2024-04-24T01:25:00.347 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.111259.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104715.log.gz 2024-04-24T01:25:00.347 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61728.log 2024-04-24T01:25:00.348 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82004.log 2024-04-24T01:25:00.348 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114464.log 2024-04-24T01:25:00.349 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52359.log 2024-04-24T01:25:00.350 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114464.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114464.log.gz 2024-04-24T01:25:00.351 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134116.log 2024-04-24T01:25:00.351 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97491.log 2024-04-24T01:25:00.352 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134116.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134116.log.gz 2024-04-24T01:25:00.353 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65391.log 2024-04-24T01:25:00.353 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111101.log 2024-04-24T01:25:00.354 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141544.log 2024-04-24T01:25:00.355 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111101.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111101.log.gz 2024-04-24T01:25:00.356 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51680.log 2024-04-24T01:25:00.356 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141544.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.141544.log.gz 2024-04-24T01:25:00.357 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97215.log 2024-04-24T01:25:00.357 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63281.log 2024-04-24T01:25:00.358 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83732.log 2024-04-24T01:25:00.359 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63281.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124539.log 2024-04-24T01:25:00.360 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.83732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127940.log 2024-04-24T01:25:00.361 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124539.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124539.log.gz 2024-04-24T01:25:00.361 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113798.log 2024-04-24T01:25:00.362 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127940.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.81899.log.gz 2024-04-24T01:25:00.362 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82109.log.gz 2024-04-24T01:25:00.362 INFO:teuthology.orchestra.run.smithi134.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127940.log.gz 2024-04-24T01:25:00.362 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59939.log.gz 2024-04-24T01:25:00.363 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77730.log.gz 2024-04-24T01:25:00.365 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65391.log.gz 2024-04-24T01:25:00.365 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.64766.log.gz 2024-04-24T01:25:00.365 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67726.log.gz 2024-04-24T01:25:00.366 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87426.log.gz 2024-04-24T01:25:00.366 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87741.log.gz 2024-04-24T01:25:00.367 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91178.log 2024-04-24T01:25:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113798.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113798.log.gz 2024-04-24T01:25:00.369 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125984.log 2024-04-24T01:25:00.369 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34836.log 2024-04-24T01:25:00.370 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125984.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125984.log.gz 2024-04-24T01:25:00.375 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117594.log 2024-04-24T01:25:00.376 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78220.log 2024-04-24T01:25:00.377 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117594.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117594.log.gz 2024-04-24T01:25:00.377 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102256.log 2024-04-24T01:25:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133761.log 2024-04-24T01:25:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.102256.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102256.log.gz 2024-04-24T01:25:00.379 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119098.log 2024-04-24T01:25:00.380 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133761.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133761.log.gz 2024-04-24T01:25:00.380 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41093.log.gz 2024-04-24T01:25:00.381 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42129.log.gz 2024-04-24T01:25:00.381 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60079.log.gz 2024-04-24T01:25:00.381 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119098.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72756.log.gz 2024-04-24T01:25:00.381 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72931.log.gz 2024-04-24T01:25:00.381 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30272.log 2024-04-24T01:25:00.383 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109794.log 2024-04-24T01:25:00.383 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117350.log 2024-04-24T01:25:00.384 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.30183.log 2024-04-24T01:25:00.384 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131862.log 2024-04-24T01:25:00.385 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139073.log 2024-04-24T01:25:00.385 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33720.log 2024-04-24T01:25:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129626.log 2024-04-24T01:25:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53465.log 2024-04-24T01:25:00.387 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55467.log 2024-04-24T01:25:00.387 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36560.log 2024-04-24T01:25:00.388 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135307.log 2024-04-24T01:25:00.388 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52557.log 2024-04-24T01:25:00.389 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33964.log 2024-04-24T01:25:00.392 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131465.log 2024-04-24T01:25:00.393 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32658.log 2024-04-24T01:25:00.393 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75035.log 2024-04-24T01:25:00.394 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74440.log 2024-04-24T01:25:00.395 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79834.log 2024-04-24T01:25:00.395 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70880.log 2024-04-24T01:25:00.396 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127404.log 2024-04-24T01:25:00.396 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135914.log 2024-04-24T01:25:00.396 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30418.log 2024-04-24T01:25:00.397 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69406.log 2024-04-24T01:25:00.397 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29096.log 2024-04-24T01:25:00.398 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65937.log 2024-04-24T01:25:00.398 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54629.log 2024-04-24T01:25:00.398 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33255.log 2024-04-24T01:25:00.398 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124225.log 2024-04-24T01:25:00.399 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112701.log 2024-04-24T01:25:00.399 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45674.log 2024-04-24T01:25:00.399 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128991.log 2024-04-24T01:25:00.400 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58008.log 2024-04-24T01:25:00.400 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116553.log 2024-04-24T01:25:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140955.log 2024-04-24T01:25:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56666.log 2024-04-24T01:25:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56828.log 2024-04-24T01:25:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105965.log 2024-04-24T01:25:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122226.log 2024-04-24T01:25:00.403 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46450.log 2024-04-24T01:25:00.403 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114920.log 2024-04-24T01:25:00.403 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45515.log 2024-04-24T01:25:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132081.log 2024-04-24T01:25:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140352.log 2024-04-24T01:25:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141379.log 2024-04-24T01:25:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36472.log 2024-04-24T01:25:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44448.log 2024-04-24T01:25:00.405 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76365.log 2024-04-24T01:25:00.405 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46485.log 2024-04-24T01:25:00.405 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57159.log 2024-04-24T01:25:00.405 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120953.log 2024-04-24T01:25:00.405 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106440.log 2024-04-24T01:25:00.406 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73351.log 2024-04-24T01:25:00.406 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126544.log 2024-04-24T01:25:00.406 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68426.log 2024-04-24T01:25:00.406 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59974.log 2024-04-24T01:25:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138306.log 2024-04-24T01:25:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121039.log 2024-04-24T01:25:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48319.log 2024-04-24T01:25:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131771.log 2024-04-24T01:25:00.408 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80009.log 2024-04-24T01:25:00.409 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141274.log 2024-04-24T01:25:00.409 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69196.log 2024-04-24T01:25:00.409 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102828.log 2024-04-24T01:25:00.409 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72126.log 2024-04-24T01:25:00.409 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140492.log 2024-04-24T01:25:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39250.log 2024-04-24T01:25:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87496.log 2024-04-24T01:25:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67271.log 2024-04-24T01:25:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91318.log 2024-04-24T01:25:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125259.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42698.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29824.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141908.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107010.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131700.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53818.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24136.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111486.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139234.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127461.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45480.log 2024-04-24T01:25:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124616.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139275.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53915.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120651.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49837.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.116441.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.61739.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107252.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40379.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31149.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95866.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62239.log 2024-04-24T01:25:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128590.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132901.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135599.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48253.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40282.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44704.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109889.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43672.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31914.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58818.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135622.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134461.log 2024-04-24T01:25:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114198.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29599.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42469.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80674.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137174.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66851.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70036.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52751.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120331.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77485.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38377.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42032.log 2024-04-24T01:25:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132830.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104075.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123323.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32135.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137219.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52103.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57717.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57071.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118323.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36492.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.48190.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29262.log 2024-04-24T01:25:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114789.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105400.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45030.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90163.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137593.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114885.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84898.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44060.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71784.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29016.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103804.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90268.log 2024-04-24T01:25:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128302.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122005.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83538.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70460.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109955.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34557.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.102369.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87811.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63526.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63316.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132103.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116418.log 2024-04-24T01:25:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140580.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35452.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51583.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105864.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77695.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93163.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118277.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40220.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90653.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128258.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134907.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106985.log 2024-04-24T01:25:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73666.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32193.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51098.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31484.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71475.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97415.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111902.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91913.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125386.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96434.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64866.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100266.log 2024-04-24T01:25:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36918.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60919.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96259.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38342.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41543.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134390.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62169.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135982.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108924.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93513.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138923.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104239.log 2024-04-24T01:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133114.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83561.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133852.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84378.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90758.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103275.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63071.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92228.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141400.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124504.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118723.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100329.log 2024-04-24T01:25:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.29774.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83493.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29175.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97651.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62970.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123544.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94835.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141494.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54691.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33313.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63421.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104590.log 2024-04-24T01:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52945.log 2024-04-24T01:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131629.log 2024-04-24T01:25:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52980.log 2024-04-24T01:25:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103645.log 2024-04-24T01:25:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64971.log 2024-04-24T01:25:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71405.log 2024-04-24T01:25:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136252.log 2024-04-24T01:25:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114429.log 2024-04-24T01:25:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.30272.log: /var/log/ceph/cluster1-client.mirror.136252.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30272.log.gz 2024-04-24T01:25:00.471 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136252.log.gz 2024-04-24T01:25:00.471 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33813.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109794.log: /var/log/ceph/cluster2-client.mirror.117350.log: /var/log/ceph/cluster1-client.mirror.2.30183.log: /var/log/ceph/cluster2-client.mirror.131862.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.131862.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139073.log: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.111259.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70425.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33720.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62910.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.129626.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49193.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50454.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96523.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.129626.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.36772.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79799.log.gz 2024-04-24T01:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53465.log: /var/log/ceph/cluster2-client.mirror.55467.log: /var/log/ceph/cluster2-client.mirror.36560.log: /var/log/ceph/cluster1-client.mirror.135307.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.135307.log.gz 2024-04-24T01:25:00.478 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52557.log: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57182.log.gz 2024-04-24T01:25:00.478 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114429.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114429.log.gz 2024-04-24T01:25:00.479 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75722.log.gz 2024-04-24T01:25:00.479 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33964.log: /var/log/ceph/cluster2-client.mirror.131465.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.131465.log.gz 2024-04-24T01:25:00.479 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32658.log: /var/log/ceph/cluster2-client.mirror.75035.log: /var/log/ceph/cluster2-client.mirror.74440.log: /var/log/ceph/cluster2-client.mirror.79834.log: /var/log/ceph/cluster2-client.mirror.70880.log: /var/log/ceph/cluster2-client.mirror.127404.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127404.log.gz 2024-04-24T01:25:00.479 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135914.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135914.log.gz 2024-04-24T01:25:00.479 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.30418.log: /var/log/ceph/cluster2-client.mirror.69406.log: /var/log/ceph/cluster2-client.admin.29096.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29096.log.gz 2024-04-24T01:25:00.479 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65937.log: /var/log/ceph/cluster2-client.mirror.54629.log: /var/log/ceph/cluster2-client.mirror.33255.log: /var/log/ceph/cluster2-client.mirror.124225.log: /var/log/ceph/cluster2-client.mirror.112701.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112701.log.gz 2024-04-24T01:25:00.479 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45674.log: /var/log/ceph/cluster2-client.mirror.128991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77100.log 2024-04-24T01:25:00.479 INFO:teuthology.orchestra.run.smithi134.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124225.log.gz 2024-04-24T01:25:00.495 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58008.log: /var/log/ceph/cluster2-client.mirror.116553.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116553.log.gz 2024-04-24T01:25:00.495 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140955.log: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109794.log.gz 2024-04-24T01:25:00.496 INFO:teuthology.orchestra.run.smithi134.stderr: 97.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117350.log.gz 2024-04-24T01:25:00.496 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56666.log: /var/log/ceph/cluster2-client.mirror.56828.log: /var/log/ceph/cluster2-client.mirror.105965.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105965.log.gz 2024-04-24T01:25:00.496 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.122226.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.29799.log.gz 2024-04-24T01:25:00.496 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122226.log.gz 2024-04-24T01:25:00.502 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46450.log: /var/log/ceph/cluster2-client.mirror.114920.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114920.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45515.log: /var/log/ceph/cluster2-client.mirror.132081.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132081.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140352.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.140352.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.141379.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141379.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36472.log: /var/log/ceph/cluster2-client.mirror.44448.log: /var/log/ceph/cluster2-client.mirror.76365.log: /var/log/ceph/cluster2-client.mirror.46485.log: /var/log/ceph/cluster2-client.mirror.57159.log: /var/log/ceph/cluster2-client.mirror.120953.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120953.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106440.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106440.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73351.log: /var/log/ceph/cluster2-client.mirror.126544.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126544.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68426.log: /var/log/ceph/cluster2-client.mirror.59974.log: /var/log/ceph/cluster2-client.mirror.138306.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.138306.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.121039.log: /var/log/ceph/cluster2-client.mirror.48319.log: /var/log/ceph/cluster2-client.mirror.131771.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131771.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80009.log: /var/log/ceph/cluster1-client.mirror.141274.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.141274.log.gz 2024-04-24T01:25:00.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69196.log: /var/log/ceph/cluster2-client.mirror.102828.log: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121039.log.gz 2024-04-24T01:25:00.511 INFO:teuthology.orchestra.run.smithi134.stderr: 96.6%/var/log/ceph/cluster2-client.mirror.72126.log: /var/log/ceph/cluster2-client.mirror.140492.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140492.log.gz 2024-04-24T01:25:00.511 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39250.log: /var/log/ceph/cluster2-client.mirror.87496.log: /var/log/ceph/cluster2-client.mirror.67271.log: -- replaced with /var/log/ceph/cluster2-client.mirror.102828.log.gz/var/log/ceph/cluster2-client.mirror.91318.log: 2024-04-24T01:25:00.511 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8%/var/log/ceph/cluster2-client.mirror.125259.log: -- replaced with /var/log/ceph/cluster2-client.mirror.101973.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86949.log 2024-04-24T01:25:00.511 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:00.511 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125259.log.gz 2024-04-24T01:25:00.544 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77100.log: /var/log/ceph/cluster2-client.mirror.42698.log: /var/log/ceph/cluster1-client.mirror.29824.log: /var/log/ceph/cluster2-client.admin.141908.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141908.log.gz 2024-04-24T01:25:00.544 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107010.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107010.log.gz 2024-04-24T01:25:00.544 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131700.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131700.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53818.log: /var/log/ceph/cluster2-client.admin.24136.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24136.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111486.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111486.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139234.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.139234.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127461.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.127461.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45480.log: /var/log/ceph/cluster2-client.mirror.124616.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124616.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139275.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.139275.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53915.log: /var/log/ceph/cluster2-client.mirror.120651.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.120651.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49837.log: /var/log/ceph/cluster1-client.mirror.1.116441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44836.log 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86949.log: /var/log/ceph/cluster2-client.mirror.0.61739.log: /var/log/ceph/cluster2-client.mirror.107252.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.107252.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40379.log: /var/log/ceph/cluster2-client.mirror.31149.log: /var/log/ceph/cluster2-client.mirror.95866.log: /var/log/ceph/cluster2-client.mirror.62239.log: /var/log/ceph/cluster2-client.mirror.128590.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128590.log.gz 2024-04-24T01:25:00.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132901.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132901.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135599.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135599.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48253.log: /var/log/ceph/cluster2-client.mirror.40282.log: /var/log/ceph/cluster2-client.mirror.44704.log: /var/log/ceph/cluster2-client.mirror.109889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89071.log 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109889.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43672.log: /var/log/ceph/cluster2-client.mirror.44836.log: /var/log/ceph/cluster2-client.mirror.31914.log: /var/log/ceph/cluster2-client.mirror.58818.log: /var/log/ceph/cluster1-client.mirror.135622.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135622.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134461.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134461.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114198.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114198.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29599.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29599.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42469.log: /var/log/ceph/cluster2-client.mirror.80674.log: /var/log/ceph/cluster1-client.mirror.137174.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137174.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66851.log: /var/log/ceph/cluster2-client.mirror.70036.log: /var/log/ceph/cluster2-client.mirror.52751.log: /var/log/ceph/cluster2-client.mirror.120331.log: /var/log/ceph/cluster2-client.mirror.77485.log: /var/log/ceph/cluster2-client.mirror.38377.log: /var/log/ceph/cluster2-client.mirror.42032.log: /var/log/ceph/cluster2-client.mirror.132830.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132830.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104075.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104075.log.gz 2024-04-24T01:25:00.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123323.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.123323.log.gz 2024-04-24T01:25:00.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32135.log: /var/log/ceph/cluster1-client.mirror.137219.log: /var/log/ceph/cluster2-client.mirror.52103.log: /var/log/ceph/cluster2-client.mirror.57717.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137219.log.gz 2024-04-24T01:25:00.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.57071.log: /var/log/ceph/cluster2-client.mirror.118323.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120331.log.gz 2024-04-24T01:25:00.547 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118323.log.gz 2024-04-24T01:25:00.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.36492.log: /var/log/ceph/cluster1-client.mirror.0.48190.log: /var/log/ceph/cluster2-client.admin.29262.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29262.log.gz 2024-04-24T01:25:00.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114789.log: /var/log/ceph/cluster2-client.mirror.105400.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105400.log.gz 2024-04-24T01:25:00.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45030.log: /var/log/ceph/cluster2-client.mirror.90163.log: /var/log/ceph/cluster2-client.mirror.137593.log: /var/log/ceph/cluster2-client.mirror.114885.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114885.log.gz 2024-04-24T01:25:00.563 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84898.log: /var/log/ceph/cluster2-client.mirror.44060.log: /var/log/ceph/cluster2-client.mirror.71784.log: /var/log/ceph/cluster2-client.admin.29016.log: /var/log/ceph/cluster2-client.mirror.103804.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29016.log.gz 2024-04-24T01:25:00.563 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90268.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103804.log.gz 2024-04-24T01:25:00.564 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.128302.log: /var/log/ceph/cluster2-client.mirror.122005.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122005.log.gz 2024-04-24T01:25:00.564 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128302.log.gz 2024-04-24T01:25:00.564 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83538.log: /var/log/ceph/cluster2-client.mirror.70460.log: /var/log/ceph/cluster2-client.mirror.109955.log: /var/log/ceph/cluster2-client.mirror.34557.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109955.log.gz 2024-04-24T01:25:00.567 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.3.102369.log: /var/log/ceph/cluster2-client.mirror.87811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132739.log 2024-04-24T01:25:00.571 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89071.log: /var/log/ceph/cluster2-client.mirror.63526.log: /var/log/ceph/cluster2-client.mirror.63316.log: /var/log/ceph/cluster2-client.mirror.132103.log: /var/log/ceph/cluster2-client.mirror.116418.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.132103.log.gz 2024-04-24T01:25:00.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140580.log: /var/log/ceph/cluster2-client.mirror.35452.log: /var/log/ceph/cluster2-client.mirror.51583.log: /var/log/ceph/cluster2-client.mirror.105864.log: /var/log/ceph/cluster2-client.mirror.77695.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105864.log.gz 2024-04-24T01:25:00.581 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93163.log: /var/log/ceph/cluster2-client.mirror.118277.log: /var/log/ceph/cluster2-client.mirror.40220.log: /var/log/ceph/cluster2-client.mirror.90653.log: /var/log/ceph/cluster2-client.mirror.128258.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118277.log.gz 2024-04-24T01:25:00.581 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128258.log.gz 2024-04-24T01:25:00.582 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134907.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134907.log.gz 2024-04-24T01:25:00.582 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.106985.log: /var/log/ceph/cluster2-client.mirror.73666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104909.log 2024-04-24T01:25:00.583 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106985.log.gz 2024-04-24T01:25:00.584 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32193.log: /var/log/ceph/cluster2-client.mirror.51098.log: /var/log/ceph/cluster2-client.mirror.31484.log: /var/log/ceph/cluster2-client.mirror.71475.log: /var/log/ceph/cluster2-client.mirror.97415.log: /var/log/ceph/cluster2-client.mirror.111902.log: /var/log/ceph/cluster2-client.mirror.91913.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.111902.log.gz 2024-04-24T01:25:00.597 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125386.log: /var/log/ceph/cluster1-client.mirror.96434.log: /var/log/ceph/cluster2-client.mirror.64866.log: /var/log/ceph/cluster2-client.mirror.100266.log: /var/log/ceph/cluster2-client.mirror.36918.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125386.log.gz 2024-04-24T01:25:00.597 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132739.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132739.log.gz 2024-04-24T01:25:00.597 INFO:teuthology.orchestra.run.smithi134.stderr: 95.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64508.log.gz 2024-04-24T01:25:00.597 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60919.log: /var/log/ceph/cluster1-client.mirror.96259.log: /var/log/ceph/cluster2-client.mirror.38342.log: /var/log/ceph/cluster2-client.mirror.41543.log: /var/log/ceph/cluster2-client.mirror.134390.log: /var/log/ceph/cluster2-client.mirror.62169.log: /var/log/ceph/cluster1-client.mirror.135982.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135982.log.gz 2024-04-24T01:25:00.597 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134390.log.gz 2024-04-24T01:25:00.597 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108924.log: /var/log/ceph/cluster2-client.mirror.93513.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108924.log.gz 2024-04-24T01:25:00.597 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138923.log: /var/log/ceph/cluster1-client.mirror.104239.log: /var/log/ceph/cluster2-client.mirror.133114.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138923.log.gz 2024-04-24T01:25:00.598 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104239.log.gz 2024-04-24T01:25:00.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83561.log: /var/log/ceph/cluster2-client.mirror.133852.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133114.log.gz 2024-04-24T01:25:00.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57426.log 2024-04-24T01:25:00.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.104909.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133852.log.gz 2024-04-24T01:25:00.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90758.log: /var/log/ceph/cluster2-client.mirror.103275.log: /var/log/ceph/cluster2-client.mirror.63071.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.103275.log.gz 2024-04-24T01:25:00.602 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92228.log: /var/log/ceph/cluster2-client.mirror.141400.log: /var/log/ceph/cluster2-client.mirror.124504.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141400.log.gz 2024-04-24T01:25:00.602 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124504.log.gz 2024-04-24T01:25:00.602 INFO:teuthology.orchestra.run.smithi134.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.104909.log.gz 2024-04-24T01:25:00.602 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118723.log: /var/log/ceph/cluster2-client.mirror.100329.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100329.log.gz 2024-04-24T01:25:00.602 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118723.log.gz 2024-04-24T01:25:00.602 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.29774.log: /var/log/ceph/cluster1-client.mirror.83493.log: /var/log/ceph/cluster2-client.admin.29175.log: /var/log/ceph/cluster2-client.mirror.97651.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29175.log.gz 2024-04-24T01:25:00.603 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62970.log: /var/log/ceph/cluster1-client.mirror.123544.log: /var/log/ceph/cluster2-client.mirror.94835.log: /var/log/ceph/cluster1-client.mirror.141494.log: /var/log/ceph/cluster2-client.mirror.54691.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141494.log.gz 2024-04-24T01:25:00.603 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33313.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123544.log.gz 2024-04-24T01:25:00.603 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69161.log.gz 2024-04-24T01:25:00.603 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89810.log.gz 2024-04-24T01:25:00.603 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85904.log.gz 2024-04-24T01:25:00.603 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63421.log: /var/log/ceph/cluster2-client.mirror.104590.log: /var/log/ceph/cluster2-client.mirror.52945.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64423.log.gz 2024-04-24T01:25:00.604 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104590.log.gz 2024-04-24T01:25:00.604 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45383.log.gz 2024-04-24T01:25:00.604 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42601.log.gz 2024-04-24T01:25:00.604 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57585.log.gz 2024-04-24T01:25:00.604 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82774.log.gz 2024-04-24T01:25:00.604 INFO:teuthology.orchestra.run.smithi134.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64331.log.gz 2024-04-24T01:25:00.604 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80852.log.gz 2024-04-24T01:25:00.605 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32228.log.gz 2024-04-24T01:25:00.605 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31670.log.gz 2024-04-24T01:25:00.605 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95971.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45418.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33871.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.36702.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34708.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.131629.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39312.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34650.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86499.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58783.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131629.log.gz 2024-04-24T01:25:00.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52980.log: /var/log/ceph/cluster2-client.mirror.103645.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103645.log.gz 2024-04-24T01:25:00.608 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64971.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52883.log.gz 2024-04-24T01:25:00.608 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80763.log.gz 2024-04-24T01:25:00.608 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94477.log.gz 2024-04-24T01:25:00.608 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71405.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119098.log.gz 2024-04-24T01:25:00.608 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39929.log.gz 2024-04-24T01:25:00.608 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.116441.log.gz 2024-04-24T01:25:00.611 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100578.log.gz 2024-04-24T01:25:00.611 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97155.log 2024-04-24T01:25:00.612 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63806.log 2024-04-24T01:25:00.613 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120864.log 2024-04-24T01:25:00.613 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83054.log 2024-04-24T01:25:00.614 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120864.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120864.log.gz 2024-04-24T01:25:00.626 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93933.log 2024-04-24T01:25:00.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126467.log 2024-04-24T01:25:00.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93933.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42221.log.gz 2024-04-24T01:25:00.643 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.138712.log.gz 2024-04-24T01:25:00.647 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74860.log.gz 2024-04-24T01:25:00.650 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116418.log.gz 2024-04-24T01:25:00.650 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54338.log 2024-04-24T01:25:00.651 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126467.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126467.log.gz 2024-04-24T01:25:00.663 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39347.log 2024-04-24T01:25:00.684 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54338.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44704.log.gz 2024-04-24T01:25:00.684 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53465.log.gz 2024-04-24T01:25:00.684 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74195.log.gz 88.3% 2024-04-24T01:25:00.684 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60919.log.gz 2024-04-24T01:25:00.684 INFO:teuthology.orchestra.run.smithi134.stderr: 74.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62970.log.gz 2024-04-24T01:25:00.684 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68356.log.gz 2024-04-24T01:25:00.685 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63281.log.gz 2024-04-24T01:25:00.686 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63316.log.gz 2024-04-24T01:25:00.686 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45286.log.gz 2024-04-24T01:25:00.686 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95992.log.gz 2024-04-24T01:25:00.686 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45674.log.gz 2024-04-24T01:25:00.686 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96034.log.gz 2024-04-24T01:25:00.686 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54338.log.gz 2024-04-24T01:25:00.686 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51583.log.gz 2024-04-24T01:25:00.686 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46450.log.gz 2024-04-24T01:25:00.686 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45224.log.gz 2024-04-24T01:25:00.688 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46159.log.gz 2024-04-24T01:25:00.688 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53915.log.gz 2024-04-24T01:25:00.688 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74545.log.gz 2024-04-24T01:25:00.688 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57523.log 2024-04-24T01:25:00.688 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45030.log.gz 2024-04-24T01:25:00.688 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60604.log.gz 2024-04-24T01:25:00.688 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44642.log.gz 2024-04-24T01:25:00.688 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39347.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44025.log.gz 2024-04-24T01:25:00.688 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44060.log.gz 2024-04-24T01:25:00.689 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.95284.log.gz 2024-04-24T01:25:00.690 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93513.log.gz 2024-04-24T01:25:00.691 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57026.log.gz 2024-04-24T01:25:00.693 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56666.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.56712.log.gz 2024-04-24T01:25:00.693 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:00.696 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140805.log 2024-04-24T01:25:00.708 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57523.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43769.log.gz 2024-04-24T01:25:00.710 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81619.log 2024-04-24T01:25:00.711 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140805.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.140805.log.gz 2024-04-24T01:25:00.726 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125305.log 2024-04-24T01:25:00.727 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81619.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33720.log.gz 2024-04-24T01:25:00.730 INFO:teuthology.orchestra.run.smithi067.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-04-24T01:25:00.731 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75035.log.gz 2024-04-24T01:25:00.732 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91493.log.gz 2024-04-24T01:25:00.732 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72441.log.gz 2024-04-24T01:25:00.732 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37857.log.gz 2024-04-24T01:25:00.733 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78220.log.gz 2024-04-24T01:25:00.734 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95824.log.gz 2024-04-24T01:25:00.734 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.89740.log.gz 2024-04-24T01:25:00.735 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59413.log.gz 2024-04-24T01:25:00.736 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90163.log.gz 2024-04-24T01:25:00.737 INFO:teuthology.orchestra.run.smithi134.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61728.log.gz 2024-04-24T01:25:00.738 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95499.log 2024-04-24T01:25:00.738 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70320.log.gz 2024-04-24T01:25:00.739 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125305.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80009.log.gz 2024-04-24T01:25:00.739 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125305.log.gz 2024-04-24T01:25:00.742 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88756.log.gz 2024-04-24T01:25:00.743 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33964.log.gz 2024-04-24T01:25:00.743 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75814.log.gz 2024-04-24T01:25:00.744 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44448.log.gz 2024-04-24T01:25:00.746 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46485.log.gz 2024-04-24T01:25:00.747 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32658.log.gz 2024-04-24T01:25:00.747 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42892.log.gz 2024-04-24T01:25:00.748 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66637.log.gz 2024-04-24T01:25:00.751 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98171.log 2024-04-24T01:25:00.751 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58008.log.gz 2024-04-24T01:25:00.752 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95499.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69196.log.gz 2024-04-24T01:25:00.752 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100668.log.gz 2024-04-24T01:25:00.752 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95695.log.gz 2024-04-24T01:25:00.753 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96236.log.gz 2024-04-24T01:25:00.753 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95632.log.gz 2024-04-24T01:25:00.756 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39250.log.gz 2024-04-24T01:25:00.756 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39347.log.gz 2024-04-24T01:25:00.757 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45480.log.gz 2024-04-24T01:25:00.758 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52557.log.gz 2024-04-24T01:25:00.758 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48319.log.gz 2024-04-24T01:25:00.758 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48253.log.gz 2024-04-24T01:25:00.758 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51680.log.gz 2024-04-24T01:25:00.759 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55502.log.gz 2024-04-24T01:25:00.759 INFO:teuthology.orchestra.run.smithi134.stderr: 76.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29824.log.gz 2024-04-24T01:25:00.761 INFO:teuthology.orchestra.run.smithi134.stderr: 73.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30934.log.gz 2024-04-24T01:25:00.762 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114789.log.gz 2024-04-24T01:25:00.762 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53818.log.gz 2024-04-24T01:25:00.762 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54629.log.gz 2024-04-24T01:25:00.763 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72126.log.gz 2024-04-24T01:25:00.763 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86324.log.gz 2024-04-24T01:25:00.772 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79834.log.gz 2024-04-24T01:25:00.772 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82004.log.gz 2024-04-24T01:25:00.774 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55855.log 2024-04-24T01:25:00.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98171.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98171.log.gz 2024-04-24T01:25:00.787 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38280.log.gz 2024-04-24T01:25:00.789 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97491.log.gz 2024-04-24T01:25:00.790 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70880.log.gz 2024-04-24T01:25:00.791 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.90653.log.gz 2024-04-24T01:25:00.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93198.log 2024-04-24T01:25:00.795 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87496.log.gz 2024-04-24T01:25:00.796 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59974.log.gz 2024-04-24T01:25:00.796 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76365.log.gz 2024-04-24T01:25:00.796 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81619.log.gz 2024-04-24T01:25:00.796 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87811.log.gz 2024-04-24T01:25:00.796 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87951.log.gz 2024-04-24T01:25:00.796 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33255.log.gz 2024-04-24T01:25:00.809 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38377.log.gz 2024-04-24T01:25:00.810 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139108.log 2024-04-24T01:25:00.813 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93198.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45515.log.gz 2024-04-24T01:25:00.813 INFO:teuthology.orchestra.run.smithi134.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.62970.log.gz 2024-04-24T01:25:00.817 INFO:teuthology.orchestra.run.smithi134.stderr: 95.5% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55855.log.gz 2024-04-24T01:25:00.818 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43672.log.gz 2024-04-24T01:25:00.818 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.128369.log.gz 2024-04-24T01:25:00.819 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62239.log.gz 2024-04-24T01:25:00.819 INFO:teuthology.orchestra.run.smithi134.stderr: 58.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30418.log.gz 2024-04-24T01:25:00.819 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33313.log.gz 2024-04-24T01:25:00.821 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91318.log.gz 2024-04-24T01:25:00.826 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140955.log.gz 2024-04-24T01:25:00.826 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92669.log 2024-04-24T01:25:00.827 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139108.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.139108.log.gz 2024-04-24T01:25:00.828 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139315.log 2024-04-24T01:25:00.828 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38474.log 2024-04-24T01:25:00.829 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139315.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.139315.log.gz 2024-04-24T01:25:00.830 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83283.log 2024-04-24T01:25:00.831 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140888.log 2024-04-24T01:25:00.832 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.128991.log.gz 2024-04-24T01:25:00.832 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105446.log 2024-04-24T01:25:00.832 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28958.log 2024-04-24T01:25:00.834 INFO:teuthology.orchestra.run.smithi134.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140888.log.gz 2024-04-24T01:25:00.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105446.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99756.log 2024-04-24T01:25:00.835 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.105446.log.gz 2024-04-24T01:25:00.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.28958.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62134.log 2024-04-24T01:25:00.835 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0%/var/log/ceph/cluster2-client.mirror.99756.log: -- replaced with /var/log/ceph/cluster2-client.admin.28958.log.gz 2024-04-24T01:25:00.835 INFO:teuthology.orchestra.run.smithi134.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99756.log.gz 2024-04-24T01:25:00.835 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76260.log 2024-04-24T01:25:00.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62134.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110793.log 2024-04-24T01:25:00.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81041.log 2024-04-24T01:25:00.838 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56878.log 2024-04-24T01:25:00.838 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110793.log: /var/log/ceph/cluster2-client.mirror.81041.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28880.log 2024-04-24T01:25:00.838 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.110793.log.gz 2024-04-24T01:25:00.838 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114275.log 2024-04-24T01:25:00.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.28880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84658.log 2024-04-24T01:25:00.839 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28880.log.gz 2024-04-24T01:25:00.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38439.log 2024-04-24T01:25:00.839 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114275.log.gz 2024-04-24T01:25:00.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136274.log 2024-04-24T01:25:00.840 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58361.log 2024-04-24T01:25:00.840 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136274.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136274.log.gz 2024-04-24T01:25:00.840 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125934.log 2024-04-24T01:25:00.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37116.log 2024-04-24T01:25:00.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125934.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125934.log.gz 2024-04-24T01:25:00.841 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65706.log 2024-04-24T01:25:00.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109013.log 2024-04-24T01:25:00.843 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65706.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139798.log 2024-04-24T01:25:00.846 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109013.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109013.log.gz 2024-04-24T01:25:00.847 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122741.log 2024-04-24T01:25:00.847 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41838.log 2024-04-24T01:25:00.847 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139798.log: 81.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139798.log.gz 2024-04-24T01:25:00.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122741.log: /var/log/ceph/cluster2-client.mirror.41838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45709.log 2024-04-24T01:25:00.849 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125127.log 2024-04-24T01:25:00.850 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.122741.log.gz 2024-04-24T01:25:00.850 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134949.log 2024-04-24T01:25:00.851 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131791.log 2024-04-24T01:25:00.851 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125127.log: /var/log/ceph/cluster2-client.mirror.134949.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125127.log.gz 2024-04-24T01:25:00.851 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77415.log 2024-04-24T01:25:00.852 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131791.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134949.log.gz 2024-04-24T01:25:00.853 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38245.log 2024-04-24T01:25:00.853 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77485.log.gz 2024-04-24T01:25:00.854 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106162.log 2024-04-24T01:25:00.854 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38245.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131791.log.gz 2024-04-24T01:25:00.854 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140532.log 2024-04-24T01:25:00.856 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.106162.log: 92.1% 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.106162.log.gz 2024-04-24T01:25:00.856 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.90758.log.gz 2024-04-24T01:25:00.857 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133264.log 2024-04-24T01:25:00.857 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94893.log 2024-04-24T01:25:00.857 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.133264.log: -- replaced with /var/log/ceph/cluster2-client.mirror.140532.log.gz 2024-04-24T01:25:00.858 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133264.log.gz 2024-04-24T01:25:00.858 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53430.log 2024-04-24T01:25:00.858 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119397.log 2024-04-24T01:25:00.858 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94893.log: /var/log/ceph/cluster2-client.mirror.53430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114310.log 2024-04-24T01:25:00.859 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80674.log.gz 2024-04-24T01:25:00.859 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118063.log 2024-04-24T01:25:00.860 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119397.log: /var/log/ceph/cluster2-client.mirror.114310.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119397.log.gz 2024-04-24T01:25:00.860 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.89071.log.gz 2024-04-24T01:25:00.860 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.114310.log.gz 2024-04-24T01:25:00.861 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96709.log 2024-04-24T01:25:00.861 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141830.log 2024-04-24T01:25:00.861 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118063.log: /var/log/ceph/cluster2-client.mirror.96709.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118063.log.gz 2024-04-24T01:25:00.861 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136094.log 2024-04-24T01:25:00.863 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38183.log 2024-04-24T01:25:00.863 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141830.log: /var/log/ceph/cluster1-client.mirror.136094.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141830.log.gz 2024-04-24T01:25:00.863 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136094.log.gz 2024-04-24T01:25:00.863 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.98400.log 2024-04-24T01:25:00.863 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104931.log 2024-04-24T01:25:00.864 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38183.log: /var/log/ceph/cluster2-client.mirror.3.98400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117308.log 2024-04-24T01:25:00.864 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84658.log.gz 2024-04-24T01:25:00.865 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104931.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93198.log.gz 2024-04-24T01:25:00.865 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31914.log.gz 2024-04-24T01:25:00.865 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32193.log.gz 2024-04-24T01:25:00.865 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32135.log.gz 2024-04-24T01:25:00.866 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51098.log.gz 2024-04-24T01:25:00.866 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68426.log.gz 2024-04-24T01:25:00.866 INFO:teuthology.orchestra.run.smithi134.stderr: 73.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36560.log.gz 2024-04-24T01:25:00.866 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35452.log.gz 2024-04-24T01:25:00.866 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57717.log.gz 2024-04-24T01:25:00.870 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96259.log.gz 2024-04-24T01:25:00.871 INFO:teuthology.orchestra.run.smithi134.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97415.log.gz 2024-04-24T01:25:00.871 INFO:teuthology.orchestra.run.smithi134.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.61739.log.gz 2024-04-24T01:25:00.872 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91178.log.gz 2024-04-24T01:25:00.872 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92228.log.gz 2024-04-24T01:25:00.878 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.98400.log.gz 2024-04-24T01:25:00.882 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96367.log 2024-04-24T01:25:00.883 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116441.log 2024-04-24T01:25:00.883 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117308.log: /var/log/ceph/cluster2-client.mirror.96367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103084.log 2024-04-24T01:25:00.885 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116441.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117308.log.gz 2024-04-24T01:25:00.887 INFO:teuthology.orchestra.run.smithi134.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116441.log.gz 2024-04-24T01:25:00.890 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141218.log 2024-04-24T01:25:00.893 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67271.log.gz 2024-04-24T01:25:00.894 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103084.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103084.log.gz 2024-04-24T01:25:00.894 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.139798.log 2024-04-24T01:25:00.895 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141218.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141218.log.gz 2024-04-24T01:25:00.895 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61339.log 2024-04-24T01:25:00.905 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.139798.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104931.log.gz 2024-04-24T01:25:00.907 INFO:teuthology.orchestra.run.smithi134.stderr: 94.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.139798.log.gz 2024-04-24T01:25:00.907 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138900.log 2024-04-24T01:25:00.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137152.log 2024-04-24T01:25:00.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101825.log 2024-04-24T01:25:00.909 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138900.log: /var/log/ceph/cluster1-client.mirror.137152.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137152.log.gz 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138900.log.gz 2024-04-24T01:25:00.909 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:00.909 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75490.log 2024-04-24T01:25:00.910 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129374.log 2024-04-24T01:25:00.911 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101825.log: /var/log/ceph/cluster2-client.mirror.75490.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70460.log.gz 2024-04-24T01:25:00.911 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.83493.log.gz 2024-04-24T01:25:00.913 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38183.log.gz 2024-04-24T01:25:00.913 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38245.log.gz 2024-04-24T01:25:00.913 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77415.log.gz 2024-04-24T01:25:00.916 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71475.log.gz 2024-04-24T01:25:00.928 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138159.log 2024-04-24T01:25:00.929 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129374.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129374.log.gz 2024-04-24T01:25:00.929 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50163.log 2024-04-24T01:25:00.930 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138159.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.138159.log.gz 2024-04-24T01:25:00.931 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35080.log 2024-04-24T01:25:00.931 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50163.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83607.log 2024-04-24T01:25:00.932 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35080.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68776.log.gz 2024-04-24T01:25:00.934 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50904.log 2024-04-24T01:25:00.935 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130986.log 2024-04-24T01:25:00.936 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51715.log 2024-04-24T01:25:00.937 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130986.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.130986.log.gz 2024-04-24T01:25:00.938 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40379.log.gz 2024-04-24T01:25:00.939 INFO:teuthology.orchestra.run.smithi134.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.35909.log.gz 2024-04-24T01:25:00.939 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41349.log 2024-04-24T01:25:00.940 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41838.log.gz 2024-04-24T01:25:00.941 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33534.log 2024-04-24T01:25:00.941 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140325.log 2024-04-24T01:25:00.944 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33534.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44836.log.gz 2024-04-24T01:25:00.945 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53430.log.gz 2024-04-24T01:25:00.945 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42032.log.gz 2024-04-24T01:25:00.945 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65937.log.gz 2024-04-24T01:25:00.945 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.93163.log.gz 2024-04-24T01:25:00.946 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57426.log.gz 2024-04-24T01:25:00.948 INFO:teuthology.orchestra.run.smithi134.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63071.log.gz 2024-04-24T01:25:00.949 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99503.log 2024-04-24T01:25:00.949 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140325.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63421.log.gz 2024-04-24T01:25:00.949 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63806.log.gz 2024-04-24T01:25:00.950 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.140325.log.gz 2024-04-24T01:25:00.950 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89495.log 2024-04-24T01:25:00.952 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99503.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69406.log.gz 2024-04-24T01:25:00.953 INFO:teuthology.orchestra.run.smithi134.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100266.log.gz 2024-04-24T01:25:00.954 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95347.log.gz 2024-04-24T01:25:00.955 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37469.log 2024-04-24T01:25:00.956 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89495.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.95413.log.gz 2024-04-24T01:25:00.957 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38439.log.gz 2024-04-24T01:25:00.958 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52359.log.gz 2024-04-24T01:25:00.958 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52394.log.gz 2024-04-24T01:25:00.958 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38474.log.gz 2024-04-24T01:25:00.958 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141934.log 2024-04-24T01:25:00.958 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52751.log.gz 2024-04-24T01:25:00.958 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51715.log.gz 2024-04-24T01:25:00.958 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56878.log.gz 2024-04-24T01:25:00.958 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56828.log.gz 2024-04-24T01:25:00.959 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81211.log 2024-04-24T01:25:00.959 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141934.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141934.log.gz 2024-04-24T01:25:00.960 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138965.log 2024-04-24T01:25:00.960 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36733.log 2024-04-24T01:25:00.961 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138965.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.138965.log.gz 2024-04-24T01:25:00.962 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108339.log 2024-04-24T01:25:00.962 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.36733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56534.log 2024-04-24T01:25:00.963 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108339.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108339.log.gz 2024-04-24T01:25:00.964 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118800.log 2024-04-24T01:25:00.965 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135397.log 2024-04-24T01:25:00.965 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97651.log.gz 2024-04-24T01:25:00.966 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118800.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.118800.log.gz 2024-04-24T01:25:00.966 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131842.log 2024-04-24T01:25:00.967 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135397.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135397.log.gz 2024-04-24T01:25:00.967 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136072.log 2024-04-24T01:25:00.968 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131842.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131842.log.gz 2024-04-24T01:25:00.968 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74965.log 2024-04-24T01:25:00.969 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136072.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136072.log.gz 2024-04-24T01:25:00.970 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98491.log 2024-04-24T01:25:00.970 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46964.log 2024-04-24T01:25:00.971 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98491.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98491.log.gz 2024-04-24T01:25:00.972 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102278.log 2024-04-24T01:25:00.972 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133143.log 2024-04-24T01:25:00.976 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.102278.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102278.log.gz 2024-04-24T01:25:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119451.log 2024-04-24T01:25:00.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133143.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133143.log.gz 2024-04-24T01:25:00.979 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43831.log 2024-04-24T01:25:00.980 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119451.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119451.log.gz 2024-04-24T01:25:00.986 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125453.log 2024-04-24T01:25:00.987 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87636.log 2024-04-24T01:25:00.988 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.125453.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125453.log.gz 2024-04-24T01:25:00.990 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97064.log 2024-04-24T01:25:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93093.log 2024-04-24T01:25:00.992 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127369.log 2024-04-24T01:25:00.993 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135121.log 2024-04-24T01:25:00.994 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127369.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.127369.log.gz 2024-04-24T01:25:00.994 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86394.log 2024-04-24T01:25:01.000 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.135121.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135121.log.gz 2024-04-24T01:25:01.001 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75633.log 2024-04-24T01:25:01.001 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141594.log 2024-04-24T01:25:01.001 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124693.log 2024-04-24T01:25:01.001 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141594.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.141594.log.gz 2024-04-24T01:25:01.001 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70001.log 2024-04-24T01:25:01.001 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124693.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124693.log.gz 2024-04-24T01:25:01.001 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35859.log 2024-04-24T01:25:01.001 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136477.log 2024-04-24T01:25:01.002 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126895.log 2024-04-24T01:25:01.003 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136477.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136477.log.gz 2024-04-24T01:25:01.003 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116351.log 2024-04-24T01:25:01.004 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126895.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.126895.log.gz 2024-04-24T01:25:01.005 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79904.log.gz 2024-04-24T01:25:01.005 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83054.log.gz 2024-04-24T01:25:01.005 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64486.log.gz 2024-04-24T01:25:01.006 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86534.log 2024-04-24T01:25:01.007 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.116351.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116351.log.gz 2024-04-24T01:25:01.007 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33627.log 2024-04-24T01:25:01.008 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42241.log 2024-04-24T01:25:01.011 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77170.log 2024-04-24T01:25:01.012 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.42241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141441.log 2024-04-24T01:25:01.013 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71615.log 2024-04-24T01:25:01.019 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141441.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141441.log.gz 2024-04-24T01:25:01.020 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46353.log 2024-04-24T01:25:01.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43346.log 2024-04-24T01:25:01.021 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140183.log 2024-04-24T01:25:01.021 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55599.log 2024-04-24T01:25:01.022 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140183.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140183.log.gz 2024-04-24T01:25:01.024 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77275.log 2024-04-24T01:25:01.025 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55599.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75490.log.gz 2024-04-24T01:25:01.025 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137567.log 2024-04-24T01:25:01.026 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133477.log 2024-04-24T01:25:01.027 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.137567.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62134.log.gz 2024-04-24T01:25:01.027 INFO:teuthology.orchestra.run.smithi134.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.137567.log.gz 2024-04-24T01:25:01.027 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62169.log.gz 2024-04-24T01:25:01.030 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29939.log 2024-04-24T01:25:01.031 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133477.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133477.log.gz 2024-04-24T01:25:01.032 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60464.log 2024-04-24T01:25:01.032 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.29939.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29939.log.gz 2024-04-24T01:25:01.036 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141469.log 2024-04-24T01:25:01.037 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70211.log 2024-04-24T01:25:01.037 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.141469.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141469.log.gz 2024-04-24T01:25:01.040 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103940.log 2024-04-24T01:25:01.041 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70211.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140580.log.gz 2024-04-24T01:25:01.041 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85239.log 2024-04-24T01:25:01.041 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53721.log 2024-04-24T01:25:01.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103940.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103940.log.gz/var/log/ceph/cluster2-client.mirror.85239.log: 2024-04-24T01:25:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119118.log 2024-04-24T01:25:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98898.log 2024-04-24T01:25:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89001.log 2024-04-24T01:25:01.045 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.119118.log: /var/log/ceph/cluster2-client.mirror.98898.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.119118.log.gz 2024-04-24T01:25:01.045 INFO:teuthology.orchestra.run.smithi134.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.139073.log.gz 2024-04-24T01:25:01.046 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62770.log 2024-04-24T01:25:01.046 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134603.log 2024-04-24T01:25:01.047 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53174.log 2024-04-24T01:25:01.047 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124924.log 2024-04-24T01:25:01.048 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134603.log: /var/log/ceph/cluster2-client.mirror.53174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59343.log 2024-04-24T01:25:01.051 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124924.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134603.log.gz 2024-04-24T01:25:01.051 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124924.log.gz 2024-04-24T01:25:01.051 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128347.log 2024-04-24T01:25:01.051 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29987.log 2024-04-24T01:25:01.052 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59343.log: /var/log/ceph/cluster1-client.mirror.128347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127622.log 2024-04-24T01:25:01.052 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.29987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94913.log 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127622.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127622.log.gz 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.128347.log.gz 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42566.log 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29013.log 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.94913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54012.log 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42566.log: /var/log/ceph/cluster2-client.admin.29013.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29013.log.gz 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129889.log 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87109.log 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54012.log: /var/log/ceph/cluster2-client.mirror.129889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81794.log 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129889.log.gz 2024-04-24T01:25:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39021.log 2024-04-24T01:25:01.057 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119540.log 2024-04-24T01:25:01.057 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90443.log 2024-04-24T01:25:01.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68671.log 2024-04-24T01:25:01.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119540.log: 88.0%/var/log/ceph/cluster2-client.mirror.90443.log: -- replaced with /var/log/ceph/cluster2-client.mirror.119540.log.gz 2024-04-24T01:25:01.059 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86639.log 2024-04-24T01:25:01.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68671.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97438.log 2024-04-24T01:25:01.059 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94115.log 2024-04-24T01:25:01.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48576.log 2024-04-24T01:25:01.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97438.log: /var/log/ceph/cluster1-client.mirror.94115.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51230.log 2024-04-24T01:25:01.060 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67691.log 2024-04-24T01:25:01.060 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123365.log 2024-04-24T01:25:01.061 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75668.log 2024-04-24T01:25:01.061 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135197.log 2024-04-24T01:25:01.062 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123365.log: /var/log/ceph/cluster2-client.mirror.75668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.81233.log 2024-04-24T01:25:01.062 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135197.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.135197.log.gz 2024-04-24T01:25:01.062 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34557.log.gz 2024-04-24T01:25:01.063 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34836.log.gz 2024-04-24T01:25:01.063 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35080.log.gz 2024-04-24T01:25:01.063 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40220.log.gz 2024-04-24T01:25:01.063 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40282.log.gz 2024-04-24T01:25:01.065 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31542.log 2024-04-24T01:25:01.065 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41349.log.gz 2024-04-24T01:25:01.065 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60464.log.gz 2024-04-24T01:25:01.065 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74440.log.gz 2024-04-24T01:25:01.066 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.81233.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74965.log.gz 2024-04-24T01:25:01.066 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49837.log.gz 2024-04-24T01:25:01.067 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84378.log.gz 2024-04-24T01:25:01.067 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57523.log.gz 2024-04-24T01:25:01.067 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92774.log 2024-04-24T01:25:01.069 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63526.log.gz 2024-04-24T01:25:01.069 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95866.log.gz 2024-04-24T01:25:01.069 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5%/var/log/ceph/cluster2-client.mirror.31542.log: -- replaced with /var/log/ceph/cluster2-client.mirror.55599.log.gz 2024-04-24T01:25:01.069 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35859.log.gz 2024-04-24T01:25:01.070 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116117.log 2024-04-24T01:25:01.071 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61959.log 2024-04-24T01:25:01.072 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53721.log.gz 2024-04-24T01:25:01.072 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116117.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116117.log.gz 2024-04-24T01:25:01.072 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39506.log 2024-04-24T01:25:01.072 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33627.log.gz 2024-04-24T01:25:01.073 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61959.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86394.log.gz 2024-04-24T01:25:01.073 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86534.log.gz 2024-04-24T01:25:01.075 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112432.log 2024-04-24T01:25:01.075 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58299.log 2024-04-24T01:25:01.076 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112432.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112432.log.gz 2024-04-24T01:25:01.080 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.29652.log 2024-04-24T01:25:01.080 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83372.log 2024-04-24T01:25:01.081 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.29652.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37469.log.gz 2024-04-24T01:25:01.082 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74090.log 2024-04-24T01:25:01.083 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.90268.log.gz 2024-04-24T01:25:01.084 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83372.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46194.log 2024-04-24T01:25:01.084 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49484.log 2024-04-24T01:25:01.091 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132407.log 2024-04-24T01:25:01.091 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139337.log 2024-04-24T01:25:01.092 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132407.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70036.log.gz 2024-04-24T01:25:01.094 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132407.log.gz 2024-04-24T01:25:01.094 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98616.log 2024-04-24T01:25:01.096 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139337.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.139337.log.gz 2024-04-24T01:25:01.096 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70250.log 2024-04-24T01:25:01.103 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96122.log 2024-04-24T01:25:01.109 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70250.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75857.log.gz 2024-04-24T01:25:01.110 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108258.log 2024-04-24T01:25:01.113 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96122.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96434.log.gz 2024-04-24T01:25:01.115 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70211.log.gz 2024-04-24T01:25:01.116 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70250.log.gz 2024-04-24T01:25:01.117 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83283.log.gz 2024-04-24T01:25:01.117 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83372.log.gz 2024-04-24T01:25:01.131 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140162.log 2024-04-24T01:25:01.131 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108258.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108258.log.gz 2024-04-24T01:25:01.133 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76260.log.gz 2024-04-24T01:25:01.137 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.137593.log.gz 2024-04-24T01:25:01.137 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61339.log.gz 2024-04-24T01:25:01.137 INFO:teuthology.orchestra.run.smithi134.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99503.log.gz 2024-04-24T01:25:01.138 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85869.log 2024-04-24T01:25:01.138 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30333.log 2024-04-24T01:25:01.139 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30315.log 2024-04-24T01:25:01.140 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.30333.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140162.log.gz 2024-04-24T01:25:01.140 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110177.log 2024-04-24T01:25:01.141 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.30315.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111035.log 2024-04-24T01:25:01.141 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110177.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141726.log 2024-04-24T01:25:01.142 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111035.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111035.log.gz 2024-04-24T01:25:01.142 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48000.log 2024-04-24T01:25:01.143 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141726.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141726.log.gz 2024-04-24T01:25:01.144 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110177.log.gz 2024-04-24T01:25:01.144 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.65805.log 2024-04-24T01:25:01.144 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87636.log.gz 2024-04-24T01:25:01.145 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73985.log 2024-04-24T01:25:01.145 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.65805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83089.log 2024-04-24T01:25:01.145 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135000.log 2024-04-24T01:25:01.145 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126930.log 2024-04-24T01:25:01.147 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.135000.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113186.log 2024-04-24T01:25:01.147 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126930.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135000.log.gz 2024-04-24T01:25:01.147 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126930.log.gz 2024-04-24T01:25:01.147 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90373.log 2024-04-24T01:25:01.147 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.113186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120408.log 2024-04-24T01:25:01.147 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90373.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113186.log.gz 2024-04-24T01:25:01.147 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88231.log 2024-04-24T01:25:01.148 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.120408.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120408.log.gz 2024-04-24T01:25:01.149 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47317.log 2024-04-24T01:25:01.149 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72231.log 2024-04-24T01:25:01.150 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111877.log 2024-04-24T01:25:01.150 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67306.log 2024-04-24T01:25:01.150 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.111877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112592.log 2024-04-24T01:25:01.151 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67306.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111877.log.gz 2024-04-24T01:25:01.151 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81689.log 2024-04-24T01:25:01.152 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.112592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123664.log 2024-04-24T01:25:01.153 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81689.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112592.log.gz 2024-04-24T01:25:01.154 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46964.log.gz 2024-04-24T01:25:01.154 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67166.log 2024-04-24T01:25:01.154 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35487.log 2024-04-24T01:25:01.154 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67166.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123664.log.gz 2024-04-24T01:25:01.156 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71685.log 2024-04-24T01:25:01.156 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52980.log.gz 2024-04-24T01:25:01.156 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48000.log.gz 2024-04-24T01:25:01.156 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134177.log 2024-04-24T01:25:01.156 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73666.log.gz 2024-04-24T01:25:01.156 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71685.log: /var/log/ceph/cluster2-client.mirror.35487.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.93093.log.gz 2024-04-24T01:25:01.156 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71580.log 2024-04-24T01:25:01.156 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134177.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31542.log.gz 2024-04-24T01:25:01.157 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50163.log.gz 2024-04-24T01:25:01.158 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134177.log.gz 2024-04-24T01:25:01.158 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69896.log 2024-04-24T01:25:01.158 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80114.log 2024-04-24T01:25:01.158 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108438.log 2024-04-24T01:25:01.159 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38536.log 2024-04-24T01:25:01.159 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108438.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87014.log 2024-04-24T01:25:01.162 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109376.log 2024-04-24T01:25:01.163 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37178.log 2024-04-24T01:25:01.165 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.109376.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109376.log.gz 2024-04-24T01:25:01.166 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76645.log 2024-04-24T01:25:01.166 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85694.log 2024-04-24T01:25:01.167 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76645.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.94893.log.gz 2024-04-24T01:25:01.167 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104470.log 2024-04-24T01:25:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85694.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96122.log.gz 2024-04-24T01:25:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69091.log 2024-04-24T01:25:01.169 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104470.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.104470.log.gz 2024-04-24T01:25:01.169 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107890.log 2024-04-24T01:25:01.169 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95519.log.gz 2024-04-24T01:25:01.169 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96985.log 2024-04-24T01:25:01.172 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.107890.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.107890.log.gz 2024-04-24T01:25:01.172 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34092.log 2024-04-24T01:25:01.172 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131607.log 2024-04-24T01:25:01.173 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34092.log: 70.3% -- replaced with /var/log/ceph/cluster2-client.admin.30333.log.gz 2024-04-24T01:25:01.173 INFO:teuthology.orchestra.run.smithi134.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.30315.log.gz 2024-04-24T01:25:01.174 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46194.log.gz 2024-04-24T01:25:01.174 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101825.log.gz 2024-04-24T01:25:01.175 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42306.log 2024-04-24T01:25:01.176 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131607.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131607.log.gz 2024-04-24T01:25:01.177 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91388.log 2024-04-24T01:25:01.177 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33534.log.gz 2024-04-24T01:25:01.178 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89285.log 2024-04-24T01:25:01.178 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91388.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37116.log.gz 2024-04-24T01:25:01.178 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37178.log.gz 2024-04-24T01:25:01.181 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130340.log 2024-04-24T01:25:01.181 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116040.log 2024-04-24T01:25:01.183 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130340.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130340.log.gz 2024-04-24T01:25:01.184 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96586.log 2024-04-24T01:25:01.184 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116040.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121380.log 2024-04-24T01:25:01.185 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116040.log.gz 2024-04-24T01:25:01.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92438.log 2024-04-24T01:25:01.186 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121380.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121380.log.gz 2024-04-24T01:25:01.186 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104369.log 2024-04-24T01:25:01.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135734.log 2024-04-24T01:25:01.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36883.log 2024-04-24T01:25:01.188 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135734.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104369.log.gz 2024-04-24T01:25:01.188 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135734.log.gz 2024-04-24T01:25:01.188 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66007.log 2024-04-24T01:25:01.188 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131536.log 2024-04-24T01:25:01.189 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110704.log 2024-04-24T01:25:01.189 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77800.log 2024-04-24T01:25:01.190 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131536.log: /var/log/ceph/cluster2-client.mirror.110704.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110704.log.gz 2024-04-24T01:25:01.190 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131536.log.gz 2024-04-24T01:25:01.190 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125814.log 2024-04-24T01:25:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141356.log 2024-04-24T01:25:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77800.log: /var/log/ceph/cluster2-client.mirror.125814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128223.log 2024-04-24T01:25:01.192 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125814.log.gz 2024-04-24T01:25:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.141356.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.141356.log.gz 2024-04-24T01:25:01.193 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112763.log 2024-04-24T01:25:01.193 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36514.log 2024-04-24T01:25:01.194 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128223.log: /var/log/ceph/cluster2-client.mirror.112763.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.128223.log.gz 2024-04-24T01:25:01.194 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.112763.log.gz 2024-04-24T01:25:01.194 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63176.log 2024-04-24T01:25:01.194 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41970.log 2024-04-24T01:25:01.196 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63176.log: /var/log/ceph/cluster2-client.mirror.36514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126355.log 2024-04-24T01:25:01.197 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114767.log 2024-04-24T01:25:01.197 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41970.log: /var/log/ceph/cluster2-client.mirror.126355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121426.log 2024-04-24T01:25:01.197 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126355.log.gz 2024-04-24T01:25:01.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.114767.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114767.log.gz 2024-04-24T01:25:01.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113875.log 2024-04-24T01:25:01.200 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131678.log 2024-04-24T01:25:01.201 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113875.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.121426.log.gz 2024-04-24T01:25:01.201 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113875.log.gz 2024-04-24T01:25:01.202 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137399.log 2024-04-24T01:25:01.202 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45092.log 2024-04-24T01:25:01.203 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131678.log: /var/log/ceph/cluster1-client.mirror.137399.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58818.log.gz 2024-04-24T01:25:01.203 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137399.log.gz 2024-04-24T01:25:01.203 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.131678.log.gz 2024-04-24T01:25:01.204 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95241.log 2024-04-24T01:25:01.204 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45092.log: 95.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31763.log 2024-04-24T01:25:01.204 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.98898.log.gz 2024-04-24T01:25:01.204 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83701.log 2024-04-24T01:25:01.204 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95241.log: /var/log/ceph/cluster2-client.mirror.31763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68881.log 2024-04-24T01:25:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.83701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90583.log 2024-04-24T01:25:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48708.log 2024-04-24T01:25:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90583.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76960.log.gz 2024-04-24T01:25:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48035.log 2024-04-24T01:25:01.206 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77100.log.gz 2024-04-24T01:25:01.206 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77170.log.gz 2024-04-24T01:25:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66427.log 2024-04-24T01:25:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48035.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.83732.log.gz 2024-04-24T01:25:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36380.log 2024-04-24T01:25:01.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134094.log 2024-04-24T01:25:01.208 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49290.log 2024-04-24T01:25:01.208 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78430.log 2024-04-24T01:25:01.208 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49290.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134094.log.gz 2024-04-24T01:25:01.211 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121786.log 2024-04-24T01:25:01.211 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31635.log 2024-04-24T01:25:01.211 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93723.log 2024-04-24T01:25:01.213 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.31635.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.121786.log.gz 2024-04-24T01:25:01.213 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65805.log 2024-04-24T01:25:01.214 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134136.log 2024-04-24T01:25:01.214 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114563.log 2024-04-24T01:25:01.215 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134136.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123796.log 2024-04-24T01:25:01.216 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114563.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134136.log.gz 2024-04-24T01:25:01.216 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66147.log 2024-04-24T01:25:01.217 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62029.log 2024-04-24T01:25:01.217 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123796.log: /var/log/ceph/cluster2-client.mirror.66147.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123796.log.gz 2024-04-24T01:25:01.218 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58602.log 2024-04-24T01:25:01.218 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31121.log 2024-04-24T01:25:01.221 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44995.log 2024-04-24T01:25:01.221 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.31121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81479.log 2024-04-24T01:25:01.221 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53527.log 2024-04-24T01:25:01.222 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44995.log: /var/log/ceph/cluster2-client.mirror.81479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43734.log 2024-04-24T01:25:01.223 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133781.log 2024-04-24T01:25:01.223 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113721.log 2024-04-24T01:25:01.224 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43734.log: /var/log/ceph/cluster2-client.mirror.133781.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57094.log.gz 2024-04-24T01:25:01.225 INFO:teuthology.orchestra.run.smithi134.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133781.log.gz 2024-04-24T01:25:01.225 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67586.log 2024-04-24T01:25:01.225 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113721.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113721.log.gz 2024-04-24T01:25:01.226 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69826.log 2024-04-24T01:25:01.227 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40414.log 2024-04-24T01:25:01.227 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88546.log 2024-04-24T01:25:01.228 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40414.log: /var/log/ceph/cluster2-client.mirror.88546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139047.log 2024-04-24T01:25:01.229 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30055.log 2024-04-24T01:25:01.230 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57114.log.gz 2024-04-24T01:25:01.230 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109356.log 2024-04-24T01:25:01.230 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139047.log.gz 2024-04-24T01:25:01.230 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.30055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132583.log 2024-04-24T01:25:01.232 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109356.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109356.log.gz 2024-04-24T01:25:01.233 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37213.log 2024-04-24T01:25:01.233 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132583.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132583.log.gz 2024-04-24T01:25:01.233 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54144.log 2024-04-24T01:25:01.234 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71370.log 2024-04-24T01:25:01.235 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.29651.log 2024-04-24T01:25:01.235 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72826.log 2024-04-24T01:25:01.235 INFO:teuthology.orchestra.run.smithi134.stderr: 94.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98616.log.gz 2024-04-24T01:25:01.236 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.29651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56340.log 2024-04-24T01:25:01.236 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38668.log 2024-04-24T01:25:01.237 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91633.log 2024-04-24T01:25:01.237 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88126.log 2024-04-24T01:25:01.238 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85659.log 2024-04-24T01:25:01.238 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107912.log 2024-04-24T01:25:01.238 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98194.log 2024-04-24T01:25:01.239 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85659.log: /var/log/ceph/cluster2-client.mirror.107912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84830.log 2024-04-24T01:25:01.246 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98194.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77275.log.gz 2024-04-24T01:25:01.247 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71405.log.gz 2024-04-24T01:25:01.247 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71370.log.gz 2024-04-24T01:25:01.248 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.90583.log.gz 2024-04-24T01:25:01.249 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96985.log.gz 2024-04-24T01:25:01.249 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88546.log.gz 2024-04-24T01:25:01.249 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76645.log.gz 2024-04-24T01:25:01.250 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92879.log 2024-04-24T01:25:01.250 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84830.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78045.log 2024-04-24T01:25:01.251 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68321.log 2024-04-24T01:25:01.255 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87601.log 2024-04-24T01:25:01.255 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92844.log 2024-04-24T01:25:01.256 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51618.log 2024-04-24T01:25:01.257 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34987.log 2024-04-24T01:25:01.261 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51618.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.48190.log.gz 2024-04-24T01:25:01.261 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47026.log 2024-04-24T01:25:01.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116739.log 2024-04-24T01:25:01.262 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67586.log.gz 2024-04-24T01:25:01.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39603.log 2024-04-24T01:25:01.264 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116739.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92669.log.gz 2024-04-24T01:25:01.264 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92774.log.gz 2024-04-24T01:25:01.273 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96013.log 2024-04-24T01:25:01.273 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34092.log.gz 2024-04-24T01:25:01.273 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34522.log.gz 2024-04-24T01:25:01.277 INFO:teuthology.orchestra.run.smithi134.stderr: 97.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116739.log.gz 2024-04-24T01:25:01.278 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39603.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74020.log 2024-04-24T01:25:01.278 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64016.log 2024-04-24T01:25:01.279 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101466.log 2024-04-24T01:25:01.280 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.30934.log 2024-04-24T01:25:01.281 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101466.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39506.log.gz 2024-04-24T01:25:01.282 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41543.log.gz 2024-04-24T01:25:01.282 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47026.log.gz 2024-04-24T01:25:01.282 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123365.log.gz 2024-04-24T01:25:01.283 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141115.log 2024-04-24T01:25:01.283 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.30934.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48035.log.gz 2024-04-24T01:25:01.284 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55661.log 2024-04-24T01:25:01.284 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.84830.log.gz 2024-04-24T01:25:01.284 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31484.log.gz 2024-04-24T01:25:01.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141115.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42698.log.gz 2024-04-24T01:25:01.285 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51230.log.gz 2024-04-24T01:25:01.285 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35487.log.gz 2024-04-24T01:25:01.286 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63176.log.gz 2024-04-24T01:25:01.286 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.141115.log.gz 2024-04-24T01:25:01.286 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69091.log.gz 2024-04-24T01:25:01.286 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89211.log 2024-04-24T01:25:01.287 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50066.log 2024-04-24T01:25:01.287 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125849.log 2024-04-24T01:25:01.289 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75455.log 2024-04-24T01:25:01.289 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125849.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125849.log.gz 2024-04-24T01:25:01.290 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45092.log.gz 2024-04-24T01:25:01.290 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73036.log 2024-04-24T01:25:01.290 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116717.log 2024-04-24T01:25:01.291 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30474.log 2024-04-24T01:25:01.294 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.116717.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116717.log.gz 2024-04-24T01:25:01.294 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124411.log 2024-04-24T01:25:01.295 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.30474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141519.log 2024-04-24T01:25:01.296 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124411.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124411.log.gz 2024-04-24T01:25:01.296 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78850.log 2024-04-24T01:25:01.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.141519.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141519.log.gz 2024-04-24T01:25:01.297 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139190.log 2024-04-24T01:25:01.298 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108096.log 2024-04-24T01:25:01.299 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139190.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139190.log.gz 2024-04-24T01:25:01.299 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56437.log 2024-04-24T01:25:01.300 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108096.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108096.log.gz 2024-04-24T01:25:01.300 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109331.log 2024-04-24T01:25:01.301 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113211.log 2024-04-24T01:25:01.303 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.109331.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109331.log.gz 2024-04-24T01:25:01.303 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76019.log 2024-04-24T01:25:01.305 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113211.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.113211.log.gz 2024-04-24T01:25:01.305 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84963.log 2024-04-24T01:25:01.306 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.76019.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51618.log.gz 2024-04-24T01:25:01.308 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53659.log 2024-04-24T01:25:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89425.log 2024-04-24T01:25:01.310 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64226.log 2024-04-24T01:25:01.311 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89425.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119805.log 2024-04-24T01:25:01.312 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.130141.log 2024-04-24T01:25:01.316 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119805.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119805.log.gz 2024-04-24T01:25:01.319 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53271.log 2024-04-24T01:25:01.320 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.130141.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93933.log.gz 2024-04-24T01:25:01.325 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46353.log.gz 2024-04-24T01:25:01.325 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54012.log.gz 2024-04-24T01:25:01.326 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54691.log.gz 2024-04-24T01:25:01.328 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91388.log.gz 2024-04-24T01:25:01.330 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124812.log 2024-04-24T01:25:01.332 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53271.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55661.log.gz 2024-04-24T01:25:01.334 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56340.log.gz 2024-04-24T01:25:01.334 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56534.log.gz 2024-04-24T01:25:01.336 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85869.log.gz 2024-04-24T01:25:01.337 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.80114.log.gz 2024-04-24T01:25:01.340 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36883.log.gz 2024-04-24T01:25:01.340 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36918.log.gz 2024-04-24T01:25:01.340 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91913.log.gz 2024-04-24T01:25:01.341 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38051.log.gz 2024-04-24T01:25:01.342 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.90443.log.gz 2024-04-24T01:25:01.342 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75668.log.gz 2024-04-24T01:25:01.343 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77800.log.gz 2024-04-24T01:25:01.344 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42469.log.gz 2024-04-24T01:25:01.345 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48708.log.gz 2024-04-24T01:25:01.346 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78045.log.gz 2024-04-24T01:25:01.346 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.140607.log 2024-04-24T01:25:01.347 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124812.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124812.log.gz 2024-04-24T01:25:01.347 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85694.log.gz 2024-04-24T01:25:01.349 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59343.log.gz 2024-04-24T01:25:01.351 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89001.log.gz 2024-04-24T01:25:01.351 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.89211.log.gz 2024-04-24T01:25:01.353 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92879.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.92844.log.gz 2024-04-24T01:25:01.354 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:01.355 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.81233.log.gz 2024-04-24T01:25:01.358 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42241.log.gz 2024-04-24T01:25:01.359 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73351.log.gz 2024-04-24T01:25:01.359 INFO:teuthology.orchestra.run.smithi134.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57071.log.gz 2024-04-24T01:25:01.359 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48417.log 2024-04-24T01:25:01.359 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73985.log.gz 2024-04-24T01:25:01.361 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.140607.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50066.log.gz 2024-04-24T01:25:01.361 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66427.log.gz 2024-04-24T01:25:01.362 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58361.log.gz 2024-04-24T01:25:01.370 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57682.log 2024-04-24T01:25:01.374 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48417.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38342.log.gz 2024-04-24T01:25:01.376 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38536.log.gz 2024-04-24T01:25:01.380 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48417.log.gz 2024-04-24T01:25:01.382 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48576.log.gz 2024-04-24T01:25:01.383 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55467.log.gz 2024-04-24T01:25:01.384 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126714.log 2024-04-24T01:25:01.385 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57682.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62029.log.gz 2024-04-24T01:25:01.386 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49484.log.gz 2024-04-24T01:25:01.387 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86639.log.gz 2024-04-24T01:25:01.388 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90373.log.gz 2024-04-24T01:25:01.390 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94835.log.gz 2024-04-24T01:25:01.392 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67166.log.gz 2024-04-24T01:25:01.394 INFO:teuthology.orchestra.run.smithi134.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.29987.log.gz 2024-04-24T01:25:01.401 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64226.log.gz 2024-04-24T01:25:01.402 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44316.log 2024-04-24T01:25:01.406 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.126714.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68881.log.gz 2024-04-24T01:25:01.406 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83089.log.gz 2024-04-24T01:25:01.407 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43346.log.gz 2024-04-24T01:25:01.407 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43831.log.gz 2024-04-24T01:25:01.410 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81794.log.gz 2024-04-24T01:25:01.416 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87601.log.gz 2024-04-24T01:25:01.416 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86949.log.gz 2024-04-24T01:25:01.417 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.89495.log.gz 2024-04-24T01:25:01.417 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.89425.log.gz 2024-04-24T01:25:01.417 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138484.log 2024-04-24T01:25:01.420 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44316.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126714.log.gz 2024-04-24T01:25:01.422 INFO:teuthology.orchestra.run.smithi134.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97064.log.gz 2024-04-24T01:25:01.426 INFO:teuthology.orchestra.run.smithi134.stderr: 81.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65805.log.gz 2024-04-24T01:25:01.428 INFO:teuthology.orchestra.run.smithi134.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.57182.log.gz 2024-04-24T01:25:01.428 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72826.log.gz 2024-04-24T01:25:01.429 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74090.log.gz 2024-04-24T01:25:01.431 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138484.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73036.log.gz 2024-04-24T01:25:01.431 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57682.log.gz 2024-04-24T01:25:01.431 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66851.log.gz 2024-04-24T01:25:01.431 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138604.log 2024-04-24T01:25:01.432 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62770.log.gz 2024-04-24T01:25:01.432 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.138484.log.gz 2024-04-24T01:25:01.432 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88441.log 2024-04-24T01:25:01.433 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138604.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.138604.log.gz 2024-04-24T01:25:01.435 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133526.log 2024-04-24T01:25:01.435 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88441.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69896.log.gz 2024-04-24T01:25:01.435 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129671.log 2024-04-24T01:25:01.435 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133526.log: 71.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.29774.log.gz 2024-04-24T01:25:01.436 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133526.log.gz 2024-04-24T01:25:01.436 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48902.log 2024-04-24T01:25:01.437 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.129671.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.129671.log.gz 2024-04-24T01:25:01.437 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119629.log 2024-04-24T01:25:01.438 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51389.log 2024-04-24T01:25:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119629.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119629.log.gz 2024-04-24T01:25:01.443 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83538.log.gz 2024-04-24T01:25:01.444 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38668.log.gz 2024-04-24T01:25:01.445 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45709.log.gz 2024-04-24T01:25:01.447 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43734.log.gz 2024-04-24T01:25:01.448 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52103.log.gz 2024-04-24T01:25:01.449 INFO:teuthology.orchestra.run.smithi134.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.30474.log.gz 2024-04-24T01:25:01.450 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81689.log.gz 2024-04-24T01:25:01.451 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84898.log.gz 2024-04-24T01:25:01.454 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123266.log 2024-04-24T01:25:01.456 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51389.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107912.log.gz 2024-04-24T01:25:01.456 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91633.log.gz 2024-04-24T01:25:01.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98773.log 2024-04-24T01:25:01.458 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123266.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123266.log.gz 2024-04-24T01:25:01.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136027.log 2024-04-24T01:25:01.459 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98773.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98773.log.gz 2024-04-24T01:25:01.459 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93478.log 2024-04-24T01:25:01.460 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136027.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.136027.log.gz 2024-04-24T01:25:01.460 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30350.log 2024-04-24T01:25:01.461 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32100.log 2024-04-24T01:25:01.461 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86906.log.gz 2024-04-24T01:25:01.470 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.30350.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.93723.log.gz 2024-04-24T01:25:01.486 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139135.log 2024-04-24T01:25:01.494 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32100.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96586.log.gz 2024-04-24T01:25:01.497 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85239.log.gz 2024-04-24T01:25:01.497 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86009.log 2024-04-24T01:25:01.498 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85659.log.gz 2024-04-24T01:25:01.498 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139135.log: 87.5% 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139135.log.gz 2024-04-24T01:25:01.498 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.88441.log.gz 2024-04-24T01:25:01.499 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53271.log.gz 2024-04-24T01:25:01.500 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53527.log.gz 2024-04-24T01:25:01.504 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74020.log.gz 2024-04-24T01:25:01.505 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.93478.log.gz 2024-04-24T01:25:01.506 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42566.log.gz 2024-04-24T01:25:01.511 INFO:teuthology.orchestra.run.smithi134.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58299.log.gz 2024-04-24T01:25:01.511 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70001.log.gz 2024-04-24T01:25:01.511 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36380.log.gz 2024-04-24T01:25:01.511 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51389.log.gz 2024-04-24T01:25:01.511 INFO:teuthology.orchestra.run.smithi134.stderr: 55.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30350.log.gz 2024-04-24T01:25:01.511 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49290.log.gz 2024-04-24T01:25:01.511 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95738.log 2024-04-24T01:25:01.512 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86009.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.36733.log.gz 2024-04-24T01:25:01.513 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37213.log.gz 2024-04-24T01:25:01.513 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94913.log.gz 2024-04-24T01:25:01.526 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114563.log.gz 2024-04-24T01:25:01.527 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136522.log 2024-04-24T01:25:01.528 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95476.log 2024-04-24T01:25:01.528 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136522.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136522.log.gz 2024-04-24T01:25:01.529 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131362.log 2024-04-24T01:25:01.529 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95476.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48902.log.gz 2024-04-24T01:25:01.530 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46543.log 2024-04-24T01:25:01.530 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131362.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131362.log.gz 2024-04-24T01:25:01.530 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117857.log 2024-04-24T01:25:01.531 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.46543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134023.log 2024-04-24T01:25:01.533 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.117857.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.117857.log.gz 2024-04-24T01:25:01.534 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38148.log 2024-04-24T01:25:01.534 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134023.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134023.log.gz 2024-04-24T01:25:01.535 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136994.log 2024-04-24T01:25:01.535 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94070.log 2024-04-24T01:25:01.536 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136994.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136994.log.gz 2024-04-24T01:25:01.536 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37407.log 2024-04-24T01:25:01.537 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.94070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30452.log 2024-04-24T01:25:01.537 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137287.log 2024-04-24T01:25:01.538 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.30452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111168.log 2024-04-24T01:25:01.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137287.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137287.log.gz 2024-04-24T01:25:01.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46062.log 2024-04-24T01:25:01.540 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.111168.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111168.log.gz 2024-04-24T01:25:01.540 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121345.log 2024-04-24T01:25:01.541 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138366.log 2024-04-24T01:25:01.542 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121345.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121345.log.gz 2024-04-24T01:25:01.542 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98514.log 2024-04-24T01:25:01.543 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138366.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.138366.log.gz 2024-04-24T01:25:01.543 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63841.log 2024-04-24T01:25:01.548 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98514.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88231.log.gz 2024-04-24T01:25:01.549 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89285.log.gz 2024-04-24T01:25:01.549 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.92438.log.gz 2024-04-24T01:25:01.565 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52945.log.gz 2024-04-24T01:25:01.566 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53174.log.gz 2024-04-24T01:25:01.569 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66147.log.gz 2024-04-24T01:25:01.569 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68321.log.gz 2024-04-24T01:25:01.569 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58167.log.gz 2024-04-24T01:25:01.569 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69826.log.gz 2024-04-24T01:25:01.571 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80817.log 2024-04-24T01:25:01.576 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63841.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108438.log.gz 2024-04-24T01:25:01.576 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39021.log.gz 2024-04-24T01:25:01.576 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44995.log.gz 2024-04-24T01:25:01.576 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110758.log 2024-04-24T01:25:01.577 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88686.log 2024-04-24T01:25:01.577 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110758.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110758.log.gz 2024-04-24T01:25:01.578 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66672.log 2024-04-24T01:25:01.579 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123460.log 2024-04-24T01:25:01.579 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52068.log 2024-04-24T01:25:01.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123460.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123460.log.gz 2024-04-24T01:25:01.580 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76470.log 2024-04-24T01:25:01.581 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130248.log 2024-04-24T01:25:01.581 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66252.log 2024-04-24T01:25:01.582 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130248.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72231.log.gz 2024-04-24T01:25:01.582 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130248.log.gz 2024-04-24T01:25:01.583 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.92523.log 2024-04-24T01:25:01.583 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116376.log 2024-04-24T01:25:01.584 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.92523.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75455.log.gz 2024-04-24T01:25:01.584 INFO:teuthology.orchestra.run.smithi134.stderr: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.30055.log.gz 2024-04-24T01:25:01.586 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86009.log.gz 2024-04-24T01:25:01.588 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136792.log 2024-04-24T01:25:01.589 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116376.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116376.log.gz 2024-04-24T01:25:01.590 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52165.log 2024-04-24T01:25:01.590 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136792.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136792.log.gz 2024-04-24T01:25:01.591 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95370.log 2024-04-24T01:25:01.591 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52165.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33441.log 2024-04-24T01:25:01.592 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131067.log 2024-04-24T01:25:01.592 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135419.log 2024-04-24T01:25:01.593 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131067.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131067.log.gz 2024-04-24T01:25:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125421.log 2024-04-24T01:25:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135419.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135419.log.gz 2024-04-24T01:25:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116396.log 2024-04-24T01:25:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125421.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125421.log.gz 2024-04-24T01:25:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92123.log 2024-04-24T01:25:01.597 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.116396.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116396.log.gz 2024-04-24T01:25:01.597 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92333.log 2024-04-24T01:25:01.598 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43928.log 2024-04-24T01:25:01.599 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83426.log 2024-04-24T01:25:01.599 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141752.log 2024-04-24T01:25:01.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66567.log 2024-04-24T01:25:01.601 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141752.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141752.log.gz 2024-04-24T01:25:01.601 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106000.log 2024-04-24T01:25:01.601 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103221.log 2024-04-24T01:25:01.602 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106000.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106000.log.gz 2024-04-24T01:25:01.603 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115846.log 2024-04-24T01:25:01.603 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103221.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.103221.log.gz 2024-04-24T01:25:01.604 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74125.log 2024-04-24T01:25:01.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.115846.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115846.log.gz 2024-04-24T01:25:01.606 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30072.log 2024-04-24T01:25:01.610 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140250.log 2024-04-24T01:25:01.610 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.30072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88266.log 2024-04-24T01:25:01.626 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138944.log 2024-04-24T01:25:01.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88266.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71615.log.gz 2024-04-24T01:25:01.634 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88126.log.gz 2024-04-24T01:25:01.635 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78850.log.gz 2024-04-24T01:25:01.638 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.88686.log.gz 2024-04-24T01:25:01.639 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40414.log.gz 2024-04-24T01:25:01.639 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133284.log 2024-04-24T01:25:01.639 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42284.log.gz 2024-04-24T01:25:01.639 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41970.log.gz 2024-04-24T01:25:01.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138944.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44316.log.gz 2024-04-24T01:25:01.640 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47317.log.gz 2024-04-24T01:25:01.640 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.138944.log.gz 81.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58602.log.gz 2024-04-24T01:25:01.640 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:01.640 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66007.log.gz 2024-04-24T01:25:01.641 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66252.log.gz 2024-04-24T01:25:01.644 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31763.log.gz 2024-04-24T01:25:01.644 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32100.log.gz 2024-04-24T01:25:01.644 INFO:teuthology.orchestra.run.smithi134.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101466.log.gz 2024-04-24T01:25:01.644 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67061.log 2024-04-24T01:25:01.644 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74125.log.gz 2024-04-24T01:25:01.645 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133284.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66672.log.gz 2024-04-24T01:25:01.645 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66567.log.gz 2024-04-24T01:25:01.645 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64866.log.gz 2024-04-24T01:25:01.646 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50904.log.gz 2024-04-24T01:25:01.646 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80817.log.gz 2024-04-24T01:25:01.646 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133284.log.gz 2024-04-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi134.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.95132.log.gz 2024-04-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87391.log 2024-04-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67061.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96013.log.gz 2024-04-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115523.log 2024-04-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36415.log 2024-04-24T01:25:01.649 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.115523.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52068.log.gz 2024-04-24T01:25:01.651 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56437.log.gz 2024-04-24T01:25:01.651 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94070.log.gz 2024-04-24T01:25:01.652 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115523.log.gz 2024-04-24T01:25:01.653 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63911.log 2024-04-24T01:25:01.653 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39215.log 2024-04-24T01:25:01.654 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41384.log 2024-04-24T01:25:01.654 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35115.log 2024-04-24T01:25:01.655 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91948.log 2024-04-24T01:25:01.656 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137491.log 2024-04-24T01:25:01.656 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.100693.log 2024-04-24T01:25:01.657 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137491.log: 55.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137491.log.gz 2024-04-24T01:25:01.657 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112572.log 2024-04-24T01:25:01.670 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.3.100693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134787.log 2024-04-24T01:25:01.673 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112572.log: 91.6% 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112572.log.gz 2024-04-24T01:25:01.673 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.92523.log.gz 2024-04-24T01:25:01.684 INFO:teuthology.orchestra.run.smithi134.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.102369.log.gz 2024-04-24T01:25:01.684 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100404.log 2024-04-24T01:25:01.685 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134787.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134787.log.gz 2024-04-24T01:25:01.686 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50419.log 2024-04-24T01:25:01.704 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97514.log 2024-04-24T01:25:01.715 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57911.log 2024-04-24T01:25:01.719 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97514.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83426.log.gz 2024-04-24T01:25:01.719 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76470.log.gz 2024-04-24T01:25:01.724 INFO:teuthology.orchestra.run.smithi134.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98194.log.gz 2024-04-24T01:25:01.725 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51909.log 2024-04-24T01:25:01.726 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37954.log 2024-04-24T01:25:01.726 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71300.log 2024-04-24T01:25:01.727 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140121.log 2024-04-24T01:25:01.727 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71300.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71685.log.gz 2024-04-24T01:25:01.728 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57391.log 2024-04-24T01:25:01.728 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140121.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87391.log.gz 2024-04-24T01:25:01.729 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140121.log.gz 2024-04-24T01:25:01.729 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136409.log 2024-04-24T01:25:01.730 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48514.log 2024-04-24T01:25:01.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136409.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136409.log.gz 2024-04-24T01:25:01.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115167.log 2024-04-24T01:25:01.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133832.log 2024-04-24T01:25:01.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115167.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115167.log.gz 2024-04-24T01:25:01.732 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137039.log 2024-04-24T01:25:01.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133832.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133832.log.gz 2024-04-24T01:25:01.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75280.log 2024-04-24T01:25:01.734 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137039.log: 86.8% 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.76019.log.gz 2024-04-24T01:25:01.734 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137039.log.gz 2024-04-24T01:25:01.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74615.log 2024-04-24T01:25:01.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114690.log 2024-04-24T01:25:01.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24229.log 2024-04-24T01:25:01.737 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114690.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65706.log.gz 2024-04-24T01:25:01.737 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114690.log.gz 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39603.log.gz 2024-04-24T01:25:01.737 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:01.738 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83194.log 2024-04-24T01:25:01.738 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.24229.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24229.log.gz 2024-04-24T01:25:01.739 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118835.log 2024-04-24T01:25:01.739 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44413.log 2024-04-24T01:25:01.740 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118835.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57391.log.gz 2024-04-24T01:25:01.741 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57911.log.gz 2024-04-24T01:25:01.742 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63911.log.gz 2024-04-24T01:25:01.746 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118835.log.gz 2024-04-24T01:25:01.747 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44254.log 2024-04-24T01:25:01.747 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96943.log 2024-04-24T01:25:01.747 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112614.log 2024-04-24T01:25:01.747 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84875.log 2024-04-24T01:25:01.753 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112614.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67691.log.gz 2024-04-24T01:25:01.753 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52165.log.gz 2024-04-24T01:25:01.754 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67061.log.gz 2024-04-24T01:25:01.754 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51909.log.gz 2024-04-24T01:25:01.754 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36415.log.gz 2024-04-24T01:25:01.754 INFO:teuthology.orchestra.run.smithi134.stderr: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.30072.log.gz 2024-04-24T01:25:01.756 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54144.log.gz 2024-04-24T01:25:01.757 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37407.log.gz 2024-04-24T01:25:01.761 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81479.log.gz 2024-04-24T01:25:01.762 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115055.log 2024-04-24T01:25:01.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44898.log 2024-04-24T01:25:01.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115055.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115055.log.gz 2024-04-24T01:25:01.789 INFO:teuthology.orchestra.run.smithi134.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.100693.log.gz 2024-04-24T01:25:01.790 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94657.log 2024-04-24T01:25:01.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68636.log 2024-04-24T01:25:01.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134503.log 2024-04-24T01:25:01.792 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108889.log 2024-04-24T01:25:01.795 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134503.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134503.log.gz 2024-04-24T01:25:01.795 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109854.log 2024-04-24T01:25:01.795 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108889.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.108889.log.gz 2024-04-24T01:25:01.795 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41155.log 2024-04-24T01:25:01.796 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.109854.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109854.log.gz 2024-04-24T01:25:01.797 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115132.log 2024-04-24T01:25:01.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99779.log 2024-04-24T01:25:01.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115132.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115132.log.gz 2024-04-24T01:25:01.800 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98975.log 2024-04-24T01:25:01.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99779.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68671.log.gz 2024-04-24T01:25:01.808 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78430.log.gz 2024-04-24T01:25:01.810 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83194.log.gz 2024-04-24T01:25:01.813 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57814.log 2024-04-24T01:25:01.814 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98975.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98975.log.gz 2024-04-24T01:25:01.816 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37954.log.gz 2024-04-24T01:25:01.817 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71300.log.gz 2024-04-24T01:25:01.818 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88266.log.gz 2024-04-24T01:25:01.824 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.92333.log.gz 2024-04-24T01:25:01.835 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105481.log 2024-04-24T01:25:01.848 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57814.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36492.log.gz 2024-04-24T01:25:01.850 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36472.log.gz 2024-04-24T01:25:01.856 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41155.log.gz 2024-04-24T01:25:01.857 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44413.log.gz 2024-04-24T01:25:01.858 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose 88.5% -- /var/log/ceph/cluster2-client.mirror.34150.log -- replaced with /var/log/ceph/cluster2-client.mirror.44254.log.gz 2024-04-24T01:25:01.859 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:01.859 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53659.log.gz 2024-04-24T01:25:01.859 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50419.log.gz 2024-04-24T01:25:01.859 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57814.log.gz 2024-04-24T01:25:01.860 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.105481.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105481.log.gz 2024-04-24T01:25:01.865 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64016.log.gz 2024-04-24T01:25:01.865 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64971.log.gz 2024-04-24T01:25:01.878 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133207.log 2024-04-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34150.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95476.log.gz 2024-04-24T01:25:01.885 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.30934.log.gz 2024-04-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135464.log 2024-04-24T01:25:01.901 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133207.log: 90.6% 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.65805.log.gz 2024-04-24T01:25:01.901 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82214.log 2024-04-24T01:25:01.901 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.133207.log.gz 2024-04-24T01:25:01.902 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135464.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135464.log.gz 2024-04-24T01:25:01.910 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67306.log.gz 2024-04-24T01:25:01.910 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98514.log.gz 2024-04-24T01:25:01.910 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70950.log 2024-04-24T01:25:01.911 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42760.log 2024-04-24T01:25:01.911 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106717.log 2024-04-24T01:25:01.912 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79414.log 2024-04-24T01:25:01.912 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33441.log.gz 2024-04-24T01:25:01.915 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106717.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91948.log.gz 2024-04-24T01:25:01.915 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.92123.log.gz 2024-04-24T01:25:01.924 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43928.log.gz 2024-04-24T01:25:01.925 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38148.log.gz 2024-04-24T01:25:01.926 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77695.log.gz 2024-04-24T01:25:01.928 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71580.log.gz 2024-04-24T01:25:01.930 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64366.log 2024-04-24T01:25:01.933 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79414.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75280.log.gz 2024-04-24T01:25:01.935 INFO:teuthology.orchestra.run.smithi134.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.140607.log.gz 2024-04-24T01:25:01.935 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47061.log 2024-04-24T01:25:01.936 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64366.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134219.log 2024-04-24T01:25:01.937 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132032.log 2024-04-24T01:25:01.939 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134219.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134219.log.gz 2024-04-24T01:25:01.940 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.112614.log.gz 2024-04-24T01:25:01.940 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110092.log 2024-04-24T01:25:01.941 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132032.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132032.log.gz 2024-04-24T01:25:01.942 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61304.log 2024-04-24T01:25:01.942 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126148.log 2024-04-24T01:25:01.943 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110092.log: /var/log/ceph/cluster2-client.mirror.61304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128924.log 2024-04-24T01:25:01.943 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110092.log.gz 2024-04-24T01:25:01.945 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.126148.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126148.log.gz 2024-04-24T01:25:01.946 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139006.log 2024-04-24T01:25:01.947 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98998.log 2024-04-24T01:25:01.947 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.128924.log: /var/log/ceph/cluster2-client.mirror.139006.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139006.log.gz 2024-04-24T01:25:01.950 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.128924.log.gz 2024-04-24T01:25:01.951 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31263.log 2024-04-24T01:25:01.951 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71265.log 2024-04-24T01:25:01.951 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98998.log: /var/log/ceph/cluster2-client.mirror.31263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.83786.log 2024-04-24T01:25:01.952 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71265.log: /var/log/ceph/cluster1-client.mirror.1.83786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133923.log 2024-04-24T01:25:01.953 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95069.log 2024-04-24T01:25:01.956 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133923.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133923.log.gz 2024-04-24T01:25:01.956 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71160.log 2024-04-24T01:25:01.956 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95675.log 2024-04-24T01:25:01.958 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68461.log 2024-04-24T01:25:01.959 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73491.log 2024-04-24T01:25:01.960 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63386.log 2024-04-24T01:25:01.960 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101704.log 2024-04-24T01:25:01.962 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97717.log 2024-04-24T01:25:01.963 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101704.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101704.log.gz 2024-04-24T01:25:01.963 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43443.log 2024-04-24T01:25:01.964 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78885.log 2024-04-24T01:25:01.965 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43443.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60814.log 2024-04-24T01:25:01.967 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50648.log 2024-04-24T01:25:01.967 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94180.log 2024-04-24T01:25:01.968 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135757.log 2024-04-24T01:25:01.969 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66532.log 2024-04-24T01:25:01.969 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135757.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135757.log.gz 2024-04-24T01:25:01.970 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95609.log 2024-04-24T01:25:01.970 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50225.log 2024-04-24T01:25:01.974 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126297.log 2024-04-24T01:25:01.975 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36560.log 2024-04-24T01:25:01.976 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126297.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126297.log.gz 2024-04-24T01:25:01.976 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113987.log 2024-04-24T01:25:01.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.36560.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41384.log.gz 2024-04-24T01:25:01.979 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44898.log.gz 2024-04-24T01:25:01.980 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31635.log.gz 2024-04-24T01:25:01.980 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42760.log.gz 2024-04-24T01:25:01.980 INFO:teuthology.orchestra.run.smithi134.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46543.log.gz 2024-04-24T01:25:01.980 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64366.log.gz 2024-04-24T01:25:01.980 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68636.log.gz 2024-04-24T01:25:01.981 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84855.log 2024-04-24T01:25:01.982 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.113987.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113987.log.gz 2024-04-24T01:25:01.983 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95198.log 2024-04-24T01:25:01.984 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136139.log 2024-04-24T01:25:01.985 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65286.log 2024-04-24T01:25:01.985 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136139.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136139.log.gz 2024-04-24T01:25:01.986 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84918.log 2024-04-24T01:25:01.986 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78745.log 2024-04-24T01:25:01.988 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45612.log 2024-04-24T01:25:01.989 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78745.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.31121.log.gz 2024-04-24T01:25:01.989 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54303.log 2024-04-24T01:25:01.989 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107552.log 2024-04-24T01:25:01.990 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115369.log 2024-04-24T01:25:01.991 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107552.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34987.log.gz 2024-04-24T01:25:01.991 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35115.log.gz 2024-04-24T01:25:01.991 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107552.log.gz 2024-04-24T01:25:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65426.log 2024-04-24T01:25:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115369.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115369.log.gz 2024-04-24T01:25:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62379.log 2024-04-24T01:25:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72301.log 2024-04-24T01:25:01.994 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136387.log 2024-04-24T01:25:01.995 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127503.log 2024-04-24T01:25:01.995 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136387.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136387.log.gz 2024-04-24T01:25:01.996 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37566.log 2024-04-24T01:25:02.002 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103431.log 2024-04-24T01:25:02.002 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.95241.log.gz 2024-04-24T01:25:02.009 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63841.log.gz 2024-04-24T01:25:02.010 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119257.log 2024-04-24T01:25:02.011 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.103431.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.103431.log.gz 2024-04-24T01:25:02.012 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95499.log.gz 2024-04-24T01:25:02.014 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39215.log.gz 2024-04-24T01:25:02.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37566.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75633.log.gz 2024-04-24T01:25:02.031 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94657.log.gz 2024-04-24T01:25:02.035 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61959.log.gz 2024-04-24T01:25:02.036 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48514.log.gz 2024-04-24T01:25:02.041 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106717.log.gz 2024-04-24T01:25:02.042 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116242.log 2024-04-24T01:25:02.043 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138414.log 2024-04-24T01:25:02.043 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116242.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119257.log.gz 2024-04-24T01:25:02.044 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116242.log.gz 2024-04-24T01:25:02.044 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136229.log 2024-04-24T01:25:02.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.138414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47705.log 2024-04-24T01:25:02.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136229.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136229.log.gz 2024-04-24T01:25:02.061 INFO:teuthology.orchestra.run.smithi134.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.100404.log.gz 2024-04-24T01:25:02.061 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32879.log 2024-04-24T01:25:02.062 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89845.log 2024-04-24T01:25:02.062 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50260.log 2024-04-24T01:25:02.063 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126602.log 2024-04-24T01:25:02.064 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124133.log 2024-04-24T01:25:02.064 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126602.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.126602.log.gz 2024-04-24T01:25:02.065 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139363.log 2024-04-24T01:25:02.066 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124133.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124133.log.gz 2024-04-24T01:25:02.066 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126218.log 2024-04-24T01:25:02.083 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89953.log 2024-04-24T01:25:02.084 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114140.log 2024-04-24T01:25:02.085 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38765.log 2024-04-24T01:25:02.086 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114140.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114140.log.gz 2024-04-24T01:25:02.086 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132152.log 2024-04-24T01:25:02.087 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130951.log 2024-04-24T01:25:02.088 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132152.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78885.log.gz 2024-04-24T01:25:02.088 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79414.log.gz 2024-04-24T01:25:02.088 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82214.log.gz 2024-04-24T01:25:02.088 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37566.log.gz 2024-04-24T01:25:02.089 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132152.log.gz 2024-04-24T01:25:02.092 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71265.log.gz 2024-04-24T01:25:02.092 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71160.log.gz 2024-04-24T01:25:02.093 INFO:teuthology.orchestra.run.smithi134.stderr: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126218.log.gz 2024-04-24T01:25:02.093 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134629.log 2024-04-24T01:25:02.095 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130951.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130951.log.gz 2024-04-24T01:25:02.095 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70670.log 2024-04-24T01:25:02.096 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134629.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134629.log.gz 2024-04-24T01:25:02.096 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59728.log 2024-04-24T01:25:02.097 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48805.log 2024-04-24T01:25:02.098 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84238.log 2024-04-24T01:25:02.098 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72196.log 2024-04-24T01:25:02.099 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134432.log 2024-04-24T01:25:02.100 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54435.log 2024-04-24T01:25:02.104 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134432.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134432.log.gz 2024-04-24T01:25:02.104 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98948.log 2024-04-24T01:25:02.105 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132294.log 2024-04-24T01:25:02.109 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98948.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61304.log.gz 2024-04-24T01:25:02.111 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47061.log.gz 2024-04-24T01:25:02.111 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65426.log.gz 2024-04-24T01:25:02.112 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70950.log.gz 2024-04-24T01:25:02.113 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47705.log.gz 2024-04-24T01:25:02.115 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73491.log.gz 2024-04-24T01:25:02.115 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29574.log 2024-04-24T01:25:02.115 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60814.log.gz 2024-04-24T01:25:02.115 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97717.log.gz 2024-04-24T01:25:02.117 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132294.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132294.log.gz 2024-04-24T01:25:02.117 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74615.log.gz 2024-04-24T01:25:02.117 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84238.log.gz 2024-04-24T01:25:02.119 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31263.log.gz 2024-04-24T01:25:02.122 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32879.log.gz 2024-04-24T01:25:02.124 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50225.log.gz 2024-04-24T01:25:02.126 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82879.log 2024-04-24T01:25:02.126 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29574.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29574.log.gz 2024-04-24T01:25:02.133 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38765.log.gz 2024-04-24T01:25:02.138 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72196.log.gz 2024-04-24T01:25:02.150 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129324.log 2024-04-24T01:25:02.162 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115826.log 2024-04-24T01:25:02.163 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129324.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129324.log.gz 2024-04-24T01:25:02.167 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99779.log.gz 2024-04-24T01:25:02.168 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28855.log 2024-04-24T01:25:02.170 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115826.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115826.log.gz 2024-04-24T01:25:02.170 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57332.log 2024-04-24T01:25:02.171 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.28855.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28855.log.gz 2024-04-24T01:25:02.171 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34801.log 2024-04-24T01:25:02.172 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57332.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.83701.log.gz 2024-04-24T01:25:02.172 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84168.log 2024-04-24T01:25:02.173 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96546.log 2024-04-24T01:25:02.174 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76540.log 2024-04-24T01:25:02.175 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41058.log 2024-04-24T01:25:02.175 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122194.log 2024-04-24T01:25:02.176 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87081.log 2024-04-24T01:25:02.177 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122194.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.122194.log.gz 2024-04-24T01:25:02.178 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56913.log 2024-04-24T01:25:02.178 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.87081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104142.log 2024-04-24T01:25:02.179 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73281.log 2024-04-24T01:25:02.180 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.104142.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104142.log.gz 2024-04-24T01:25:02.181 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121094.log 2024-04-24T01:25:02.181 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73281.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101124.log 2024-04-24T01:25:02.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.121094.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121094.log.gz 2024-04-24T01:25:02.183 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108593.log 2024-04-24T01:25:02.184 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101124.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89845.log.gz 2024-04-24T01:25:02.184 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89953.log.gz 2024-04-24T01:25:02.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66781.log 2024-04-24T01:25:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108593.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.108593.log.gz 2024-04-24T01:25:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82354.log 2024-04-24T01:25:02.201 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41578.log 2024-04-24T01:25:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39700.log 2024-04-24T01:25:02.203 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.41578.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42795.log 2024-04-24T01:25:02.203 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39700.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123699.log 2024-04-24T01:25:02.204 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.42795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50551.log 2024-04-24T01:25:02.205 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123699.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123699.log.gz 2024-04-24T01:25:02.206 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30021.log 2024-04-24T01:25:02.206 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70670.log.gz 2024-04-24T01:25:02.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50128.log 2024-04-24T01:25:02.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.30021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111506.log 2024-04-24T01:25:02.208 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50128.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107322.log 2024-04-24T01:25:02.211 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.111506.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111506.log.gz 2024-04-24T01:25:02.211 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123181.log 2024-04-24T01:25:02.212 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107322.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.140250.log.gz 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107322.log.gz 2024-04-24T01:25:02.212 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:02.212 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48805.log.gz 2024-04-24T01:25:02.212 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.119805.log 2024-04-24T01:25:02.213 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.123181.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123181.log.gz 2024-04-24T01:25:02.215 INFO:teuthology.orchestra.run.smithi134.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.95370.log.gz 2024-04-24T01:25:02.217 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63386.log.gz 2024-04-24T01:25:02.222 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44510.log 2024-04-24T01:25:02.230 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.3.119805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139499.log 2024-04-24T01:25:02.231 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136972.log 2024-04-24T01:25:02.232 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139499.log: 81.3% -- replaced with /var/log/ceph/cluster2-client.mirror.139499.log.gz 2024-04-24T01:25:02.232 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72301.log.gz 2024-04-24T01:25:02.251 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36101.log 2024-04-24T01:25:02.251 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136972.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136972.log.gz 2024-04-24T01:25:02.257 INFO:teuthology.orchestra.run.smithi134.stderr: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57136.log.gz 2024-04-24T01:25:02.265 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97923.log 2024-04-24T01:25:02.278 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36101.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74755.log 2024-04-24T01:25:02.279 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97923.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97923.log.gz 2024-04-24T01:25:02.282 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127503.log.gz 2024-04-24T01:25:02.282 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140907.log 2024-04-24T01:25:02.283 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132881.log 2024-04-24T01:25:02.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140907.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.140907.log.gz 2024-04-24T01:25:02.284 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65076.log 2024-04-24T01:25:02.285 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132881.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.132881.log.gz 2024-04-24T01:25:02.285 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34801.log.gz 2024-04-24T01:25:02.286 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80639.log 2024-04-24T01:25:02.286 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54109.log 2024-04-24T01:25:02.287 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111429.log 2024-04-24T01:25:02.288 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94936.log 2024-04-24T01:25:02.289 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111429.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111429.log.gz 2024-04-24T01:25:02.290 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136499.log 2024-04-24T01:25:02.290 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94936.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43575.log 2024-04-24T01:25:02.291 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136499.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136499.log.gz 2024-04-24T01:25:02.291 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84168.log.gz 2024-04-24T01:25:02.293 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133974.log 2024-04-24T01:25:02.294 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65286.log.gz 2024-04-24T01:25:02.294 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84028.log 2024-04-24T01:25:02.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133974.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50128.log.gz 2024-04-24T01:25:02.297 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133974.log.gz 2024-04-24T01:25:02.297 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50551.log.gz 2024-04-24T01:25:02.297 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43381.log 2024-04-24T01:25:02.297 INFO:teuthology.orchestra.run.smithi134.stderr: 95.3% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68461.log.gz 2024-04-24T01:25:02.297 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.1.130141.log.gz 2024-04-24T01:25:02.297 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84028.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50648.log.gz 2024-04-24T01:25:02.297 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92018.log 2024-04-24T01:25:02.298 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47158.log 2024-04-24T01:25:02.303 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92018.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65076.log.gz 2024-04-24T01:25:02.306 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115624.log 2024-04-24T01:25:02.307 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94511.log 2024-04-24T01:25:02.308 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115624.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115624.log.gz 2024-04-24T01:25:02.311 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139475.log 2024-04-24T01:25:02.311 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135020.log 2024-04-24T01:25:02.316 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139475.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.139475.log.gz 2024-04-24T01:25:02.317 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.138414.log.gz 2024-04-24T01:25:02.317 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85484.log 2024-04-24T01:25:02.317 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56793.log 2024-04-24T01:25:02.318 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.135020.log: /var/log/ceph/cluster2-client.mirror.85484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101317.log 2024-04-24T01:25:02.318 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96143.log 2024-04-24T01:25:02.319 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.135020.log.gz 2024-04-24T01:25:02.319 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101317.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101317.log.gz 2024-04-24T01:25:02.320 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94158.log 2024-04-24T01:25:02.320 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139681.log 2024-04-24T01:25:02.320 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118758.log 2024-04-24T01:25:02.321 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.94158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135374.log 2024-04-24T01:25:02.321 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139681.log: /var/log/ceph/cluster2-client.mirror.118758.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118758.log.gz 2024-04-24T01:25:02.323 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99526.log 2024-04-24T01:25:02.323 INFO:teuthology.orchestra.run.smithi134.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.119805.log.gz 2024-04-24T01:25:02.323 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135374.log: 86.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.135374.log.gz 2024-04-24T01:25:02.323 INFO:teuthology.orchestra.run.smithi134.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.36676.log 2024-04-24T01:25:02.323 INFO:teuthology.orchestra.run.smithi134.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.139681.log.gz 2024-04-24T01:25:02.325 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.99526.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78745.log.gz 2024-04-24T01:25:02.326 INFO:teuthology.orchestra.run.smithi134.stderr: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99526.log.gz 2024-04-24T01:25:02.326 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36101.log.gz 2024-04-24T01:25:02.326 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24371.log 2024-04-24T01:25:02.327 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134574.log 2024-04-24T01:25:02.327 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.36676.log: /var/log/ceph/cluster2-client.admin.24371.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24371.log.gz 2024-04-24T01:25:02.328 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40996.log 2024-04-24T01:25:02.328 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136049.log 2024-04-24T01:25:02.329 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134574.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111235.log 2024-04-24T01:25:02.329 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40996.log: 95.5%/var/log/ceph/cluster1-client.mirror.136049.log: -- replaced with /var/log/ceph/cluster2-client.mirror.98948.log.gz 2024-04-24T01:25:02.329 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136049.log.gz 2024-04-24T01:25:02.329 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73701.log 2024-04-24T01:25:02.329 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95845.log 2024-04-24T01:25:02.330 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111235.log: /var/log/ceph/cluster2-client.mirror.73701.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134574.log.gz 2024-04-24T01:25:02.330 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49158.log 2024-04-24T01:25:02.330 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127481.log 2024-04-24T01:25:02.330 INFO:teuthology.orchestra.run.smithi134.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.101124.log.gz 2024-04-24T01:25:02.330 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95845.log: /var/log/ceph/cluster2-client.mirror.49158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73596.log 2024-04-24T01:25:02.331 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.127481.log: /var/log/ceph/cluster2-client.mirror.60709.log 2024-04-24T01:25:02.331 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35638.log 2024-04-24T01:25:02.332 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63946.log 2024-04-24T01:25:02.332 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60709.log: /var/log/ceph/cluster2-client.mirror.35638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93058.log 2024-04-24T01:25:02.332 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141856.log 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.93058.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127481.log.gz 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47220.log 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141856.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141856.log.gz 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115090.log 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136544.log 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47220.log: /var/log/ceph/cluster2-client.mirror.115090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78994.log 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136544.log: gzip 87.9% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.115090.log.gz --verbose 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr: -- /var/log/ceph/cluster2-client.mirror.62274.log 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136544.log.gz 2024-04-24T01:25:02.334 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122040.log 2024-04-24T01:25:02.336 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78994.log: /var/log/ceph/cluster2-client.mirror.62274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140377.log 2024-04-24T01:25:02.336 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122040.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106085.log 2024-04-24T01:25:02.336 INFO:teuthology.orchestra.run.smithi134.stderr: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36514.log.gz 2024-04-24T01:25:02.336 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.140377.log: -- replaced with /var/log/ceph/cluster2-client.mirror.122040.log.gz 2024-04-24T01:25:02.336 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35301.log 2024-04-24T01:25:02.336 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133619.log 2024-04-24T01:25:02.336 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106085.log: /var/log/ceph/cluster2-client.mirror.35301.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106085.log.gz 2024-04-24T01:25:02.338 INFO:teuthology.orchestra.run.smithi134.stderr: 91.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82949.log 2024-04-24T01:25:02.338 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.140377.log.gz 2024-04-24T01:25:02.338 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81018.log 2024-04-24T01:25:02.338 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133619.log.gz/var/log/ceph/cluster2-client.mirror.82949.log: 2024-04-24T01:25:02.338 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60884.log 2024-04-24T01:25:02.339 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91703.log 2024-04-24T01:25:02.339 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60884.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55114.log 2024-04-24T01:25:02.339 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57273.log 2024-04-24T01:25:02.339 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118538.log 2024-04-24T01:25:02.340 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.57273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90863.log 2024-04-24T01:25:02.340 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.29824.log 2024-04-24T01:25:02.340 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118538.log: /var/log/ceph/cluster2-client.mirror.90863.log: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.111235.log.gz 2024-04-24T01:25:02.341 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95950.log 2024-04-24T01:25:02.341 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.3.29824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89106.log 2024-04-24T01:25:02.341 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132859.log 2024-04-24T01:25:02.341 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122961.log 2024-04-24T01:25:02.341 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89106.log: /var/log/ceph/cluster2-client.mirror.132859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110422.log 2024-04-24T01:25:02.342 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.122961.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125764.log 2024-04-24T01:25:02.342 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40511.log 2024-04-24T01:25:02.342 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132859.log.gz 2024-04-24T01:25:02.342 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.110422.log: /var/log/ceph/cluster2-client.mirror.125764.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125764.log.gz 2024-04-24T01:25:02.343 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55405.log 2024-04-24T01:25:02.343 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40511.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122961.log.gz 2024-04-24T01:25:02.343 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110422.log.gz 2024-04-24T01:25:02.344 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78150.log 2024-04-24T01:25:02.344 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114496.log 2024-04-24T01:25:02.344 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126669.log 2024-04-24T01:25:02.345 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.114496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141094.log 2024-04-24T01:25:02.345 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114496.log.gz 2024-04-24T01:25:02.346 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.126669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96186.log 2024-04-24T01:25:02.346 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.141094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101246.log 2024-04-24T01:25:02.346 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126669.log.gz 2024-04-24T01:25:02.346 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141094.log.gz 2024-04-24T01:25:02.346 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135352.log 2024-04-24T01:25:02.347 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95155.log 2024-04-24T01:25:02.347 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51874.log 2024-04-24T01:25:02.347 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135352.log.gz 2024-04-24T01:25:02.348 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59658.log 2024-04-24T01:25:02.349 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121806.log 2024-04-24T01:25:02.349 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59658.log: /var/log/ceph/cluster2-client.mirror.51874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137062.log 2024-04-24T01:25:02.349 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.121806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110940.log 2024-04-24T01:25:02.349 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137062.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54303.log.gz 2024-04-24T01:25:02.349 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137062.log.gz 2024-04-24T01:25:02.349 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42264.log.gz 2024-04-24T01:25:02.350 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92018.log.gz 2024-04-24T01:25:02.350 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80324.log 2024-04-24T01:25:02.351 INFO:teuthology.orchestra.run.smithi134.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121806.log.gz 2024-04-24T01:25:02.359 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134653.log 2024-04-24T01:25:02.360 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75834.log 2024-04-24T01:25:02.360 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87846.log 2024-04-24T01:25:02.361 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134653.log: /var/log/ceph/cluster1-client.mirror.75834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32972.log 2024-04-24T01:25:02.361 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77625.log 2024-04-24T01:25:02.362 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66217.log 2024-04-24T01:25:02.363 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77625.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134653.log.gz 2024-04-24T01:25:02.363 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114387.log 2024-04-24T01:25:02.363 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50357.log 2024-04-24T01:25:02.364 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115478.log 2024-04-24T01:25:02.364 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114387.log: /var/log/ceph/cluster2-client.mirror.50357.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114387.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48611.log 2024-04-24T01:25:02.365 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:02.365 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.115478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78115.log 2024-04-24T01:25:02.365 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115478.log.gz 2024-04-24T01:25:02.366 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48611.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59728.log.gz 2024-04-24T01:25:02.374 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95609.log.gz 2024-04-24T01:25:02.374 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83581.log 2024-04-24T01:25:02.375 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95908.log 2024-04-24T01:25:02.375 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78115.log: /var/log/ceph/cluster1-client.mirror.83581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129651.log 2024-04-24T01:25:02.375 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91073.log 2024-04-24T01:25:02.376 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95908.log: /var/log/ceph/cluster2-client.mirror.129651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67831.log 2024-04-24T01:25:02.377 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139423.log 2024-04-24T01:25:02.377 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67831.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129651.log.gz 2024-04-24T01:25:02.378 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97767.log 2024-04-24T01:25:02.378 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.139423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69581.log 2024-04-24T01:25:02.379 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.97767.log: -- replaced with /var/log/ceph/cluster1-client.mirror.139423.log.gz 2024-04-24T01:25:02.379 INFO:teuthology.orchestra.run.smithi134.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97767.log.gz 2024-04-24T01:25:02.379 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134807.log 2024-04-24T01:25:02.379 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59518.log 2024-04-24T01:25:02.380 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96164.log 2024-04-24T01:25:02.382 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134807.log: /var/log/ceph/cluster2-client.mirror.59518.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134807.log.gz 2024-04-24T01:25:02.383 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43443.log.gz 2024-04-24T01:25:02.383 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43381.log.gz 2024-04-24T01:25:02.383 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121694.log 2024-04-24T01:25:02.383 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67901.log 2024-04-24T01:25:02.384 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.96434.log 2024-04-24T01:25:02.384 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121694.log: 88.2%/var/log/ceph/cluster2-client.mirror.67901.log: -- replaced with /var/log/ceph/cluster2-client.mirror.121694.log.gz 2024-04-24T01:25:02.384 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29652.log 2024-04-24T01:25:02.385 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.96434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112378.log 2024-04-24T01:25:02.385 INFO:teuthology.orchestra.run.smithi134.stderr: 69.6% -- replaced with /var/log/ceph/cluster1-client.admin.30021.log.gz 2024-04-24T01:25:02.387 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.29652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107813.log 2024-04-24T01:25:02.387 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112378.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112378.log.gz 2024-04-24T01:25:02.387 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36803.log 2024-04-24T01:25:02.387 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107813.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107813.log.gz 2024-04-24T01:25:02.387 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.96736.log 2024-04-24T01:25:02.387 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46062.log.gz 2024-04-24T01:25:02.389 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.36803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29095.log 2024-04-24T01:25:02.389 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.96736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75579.log 2024-04-24T01:25:02.389 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29095.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29095.log.gz 2024-04-24T01:25:02.390 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85449.log 2024-04-24T01:25:02.390 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117251.log 2024-04-24T01:25:02.390 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110669.log 2024-04-24T01:25:02.390 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117251.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117251.log.gz 2024-04-24T01:25:02.391 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110669.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110669.log.gz 2024-04-24T01:25:02.391 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130059.log 2024-04-24T01:25:02.392 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83518.log 2024-04-24T01:25:02.392 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130059.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130059.log.gz 2024-04-24T01:25:02.393 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49387.log 2024-04-24T01:25:02.393 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81374.log 2024-04-24T01:25:02.393 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108663.log 2024-04-24T01:25:02.394 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49387.log: /var/log/ceph/cluster2-client.mirror.81374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30038.log 2024-04-24T01:25:02.394 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108663.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108663.log.gz 2024-04-24T01:25:02.395 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64794.log 2024-04-24T01:25:02.396 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78994.log.gz 2024-04-24T01:25:02.397 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82354.log.gz 2024-04-24T01:25:02.398 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.30038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59553.log 2024-04-24T01:25:02.398 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64794.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62379.log.gz 2024-04-24T01:25:02.399 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114747.log 2024-04-24T01:25:02.399 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87531.log 2024-04-24T01:25:02.400 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114747.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114747.log.gz 2024-04-24T01:25:02.402 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133335.log 2024-04-24T01:25:02.403 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139255.log 2024-04-24T01:25:02.404 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133335.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133335.log.gz 2024-04-24T01:25:02.404 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134836.log 2024-04-24T01:25:02.405 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132668.log 2024-04-24T01:25:02.406 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134836.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.139255.log.gz 2024-04-24T01:25:02.406 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64443.log 2024-04-24T01:25:02.406 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134836.log.gz 2024-04-24T01:25:02.407 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69511.log 2024-04-24T01:25:02.408 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132668.log: /var/log/ceph/cluster1-client.mirror.64443.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.132668.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62805.log 2024-04-24T01:25:02.408 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:02.408 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133165.log 2024-04-24T01:25:02.409 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53042.log 2024-04-24T01:25:02.409 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68531.log 2024-04-24T01:25:02.409 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133165.log: 88.2%/var/log/ceph/cluster2-client.mirror.53042.log: -- replaced with /var/log/ceph/cluster2-client.mirror.133165.log.gz 2024-04-24T01:25:02.409 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74405.log 2024-04-24T01:25:02.410 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127035.log 2024-04-24T01:25:02.411 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74405.log: /var/log/ceph/cluster2-client.mirror.127035.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118538.log.gz 2024-04-24T01:25:02.411 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64261.log 2024-04-24T01:25:02.411 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.127035.log.gz 2024-04-24T01:25:02.412 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57779.log 2024-04-24T01:25:02.412 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76540.log.gz 2024-04-24T01:25:02.413 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80044.log 2024-04-24T01:25:02.413 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30298.log 2024-04-24T01:25:02.413 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80044.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78115.log.gz 2024-04-24T01:25:02.414 INFO:teuthology.orchestra.run.smithi134.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.83786.log.gz 2024-04-24T01:25:02.414 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88056.log 2024-04-24T01:25:02.414 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85484.log.gz 2024-04-24T01:25:02.414 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85449.log.gz 2024-04-24T01:25:02.415 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.30298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68251.log 2024-04-24T01:25:02.415 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47511.log 2024-04-24T01:25:02.415 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91073.log.gz 2024-04-24T01:25:02.418 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132952.log 2024-04-24T01:25:02.418 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104647.log 2024-04-24T01:25:02.418 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121195.log 2024-04-24T01:25:02.419 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132952.log: /var/log/ceph/cluster2-client.mirror.104647.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135554.log 2024-04-24T01:25:02.419 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.132952.log.gz 2024-04-24T01:25:02.419 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80998.log 2024-04-24T01:25:02.420 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135554.log: 92.2% 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135554.log.gz 2024-04-24T01:25:02.420 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.104647.log.gz 2024-04-24T01:25:02.420 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84940.log 2024-04-24T01:25:02.421 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103744.log 2024-04-24T01:25:02.422 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80998.log: /var/log/ceph/cluster2-client.mirror.84940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96920.log 2024-04-24T01:25:02.422 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36447.log 2024-04-24T01:25:02.425 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103744.log: /var/log/ceph/cluster2-client.mirror.96920.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98998.log.gz 2024-04-24T01:25:02.426 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101727.log 2024-04-24T01:25:02.426 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.36447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75757.log 2024-04-24T01:25:02.438 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117780.log 2024-04-24T01:25:02.446 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75757.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84028.log.gz 2024-04-24T01:25:02.446 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95026.log 2024-04-24T01:25:02.447 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117780.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117780.log.gz 2024-04-24T01:25:02.449 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42795.log.gz 2024-04-24T01:25:02.449 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50260.log.gz 2024-04-24T01:25:02.450 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.36676.log.gz 2024-04-24T01:25:02.450 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64261.log.gz 2024-04-24T01:25:02.450 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103744.log.gz 2024-04-24T01:25:02.451 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124981.log 2024-04-24T01:25:02.451 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65531.log 2024-04-24T01:25:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113833.log 2024-04-24T01:25:02.453 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124981.log: /var/log/ceph/cluster2-client.mirror.65531.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124981.log.gz 2024-04-24T01:25:02.454 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109772.log 2024-04-24T01:25:02.455 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113833.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113833.log.gz 2024-04-24T01:25:02.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98062.log 2024-04-24T01:25:02.457 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.109772.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109772.log.gz 2024-04-24T01:25:02.458 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56472.log 2024-04-24T01:25:02.458 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98062.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.36803.log.gz 2024-04-24T01:25:02.459 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43575.log.gz 2024-04-24T01:25:02.468 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84855.log.gz 2024-04-24T01:25:02.470 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48611.log.gz 2024-04-24T01:25:02.472 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51874.log.gz 2024-04-24T01:25:02.475 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116207.log 2024-04-24T01:25:02.475 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56472.log: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30038.log.gz 2024-04-24T01:25:02.476 INFO:teuthology.orchestra.run.smithi134.stderr: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30452.log.gz 2024-04-24T01:25:02.477 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54109.log.gz 2024-04-24T01:25:02.479 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110940.log.gz 2024-04-24T01:25:02.484 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95845.log.gz 2024-04-24T01:25:02.490 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136634.log 2024-04-24T01:25:02.491 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.116207.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116207.log.gz 2024-04-24T01:25:02.492 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40670.log 2024-04-24T01:25:02.493 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136634.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136634.log.gz 2024-04-24T01:25:02.493 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33999.log 2024-04-24T01:25:02.493 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103722.log 2024-04-24T01:25:02.494 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40670.log: /var/log/ceph/cluster2-client.mirror.33999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54885.log 2024-04-24T01:25:02.495 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135239.log 2024-04-24T01:25:02.495 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.103722.log: /var/log/ceph/cluster2-client.mirror.54885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47740.log 2024-04-24T01:25:02.496 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135239.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135239.log.gz 2024-04-24T01:25:02.496 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71986.log 2024-04-24T01:25:02.497 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103722.log.gz 2024-04-24T01:25:02.497 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125023.log 2024-04-24T01:25:02.498 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24205.log 2024-04-24T01:25:02.502 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.125023.log: /var/log/ceph/cluster2-client.mirror.47740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136724.log 2024-04-24T01:25:02.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.24205.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24205.log.gz 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121195.log.gz 2024-04-24T01:25:02.503 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:02.504 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137084.log 2024-04-24T01:25:02.505 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136724.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136724.log.gz 2024-04-24T01:25:02.505 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.58602.log 2024-04-24T01:25:02.505 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95738.log.gz 2024-04-24T01:25:02.507 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137084.log: 86.9% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80044.log.gz 2024-04-24T01:25:02.507 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137084.log.gz 2024-04-24T01:25:02.507 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82949.log.gz 2024-04-24T01:25:02.507 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65531.log.gz 2024-04-24T01:25:02.507 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88056.log.gz 2024-04-24T01:25:02.508 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72616.log 2024-04-24T01:25:02.508 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.89106.log.gz 2024-04-24T01:25:02.508 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.58602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32472.log 2024-04-24T01:25:02.509 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64398.log 2024-04-24T01:25:02.509 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52592.log 2024-04-24T01:25:02.509 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135163.log 2024-04-24T01:25:02.511 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113554.log 2024-04-24T01:25:02.511 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130870.log 2024-04-24T01:25:02.512 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.135163.log: /var/log/ceph/cluster2-client.mirror.113554.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113554.log.gz 2024-04-24T01:25:02.513 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.135163.log.gz 2024-04-24T01:25:02.516 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135509.log 2024-04-24T01:25:02.516 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130870.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130870.log.gz 2024-04-24T01:25:02.516 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61409.log 2024-04-24T01:25:02.516 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135509.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135509.log.gz 2024-04-24T01:25:02.516 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50613.log 2024-04-24T01:25:02.516 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61409.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76890.log 2024-04-24T01:25:02.517 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89320.log 2024-04-24T01:25:02.517 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50613.log: /var/log/ceph/cluster2-client.mirror.76890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34278.log 2024-04-24T01:25:02.517 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67516.log 2024-04-24T01:25:02.518 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.89320.log: /var/log/ceph/cluster2-client.mirror.34278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134682.log 2024-04-24T01:25:02.518 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139025.log 2024-04-24T01:25:02.519 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67516.log: /var/log/ceph/cluster2-client.mirror.134682.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134682.log.gz 2024-04-24T01:25:02.519 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40961.log 2024-04-24T01:25:02.520 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133568.log 2024-04-24T01:25:02.520 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139025.log: /var/log/ceph/cluster2-client.mirror.40961.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.139025.log.gz 2024-04-24T01:25:02.521 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106142.log 2024-04-24T01:25:02.521 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102369.log 2024-04-24T01:25:02.524 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133568.log: /var/log/ceph/cluster2-client.mirror.106142.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39700.log.gz 2024-04-24T01:25:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133568.log.gz 2024-04-24T01:25:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40670.log.gz 2024-04-24T01:25:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40996.log.gz 2024-04-24T01:25:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44510.log.gz 2024-04-24T01:25:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41058.log.gz 2024-04-24T01:25:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40961.log.gz 2024-04-24T01:25:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47158.log.gz 2024-04-24T01:25:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47220.log.gz 2024-04-24T01:25:02.526 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112467.log 2024-04-24T01:25:02.527 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.102369.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106142.log.gz 2024-04-24T01:25:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102369.log.gz 2024-04-24T01:25:02.529 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81829.log 2024-04-24T01:25:02.530 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106405.log 2024-04-24T01:25:02.531 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112467.log: /var/log/ceph/cluster2-client.mirror.81829.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80906.log 2024-04-24T01:25:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.112467.log.gz 2024-04-24T01:25:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.139363.log.gz 2024-04-24T01:25:02.531 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117679.log 2024-04-24T01:25:02.533 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80906.log: /var/log/ceph/cluster2-client.mirror.106405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135847.log 2024-04-24T01:25:02.533 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106405.log.gz 2024-04-24T01:25:02.533 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98666.log 2024-04-24T01:25:02.533 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135847.log: 88.0% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72616.log.gz 2024-04-24T01:25:02.533 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.117679.log.gz 2024-04-24T01:25:02.533 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135847.log.gz 2024-04-24T01:25:02.533 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62484.log 2024-04-24T01:25:02.534 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111845.log 2024-04-24T01:25:02.534 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35731.log 2024-04-24T01:25:02.534 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111845.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73281.log.gz 2024-04-24T01:25:02.535 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111845.log.gz 2024-04-24T01:25:02.536 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73701.log.gz 2024-04-24T01:25:02.536 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74405.log.gz 2024-04-24T01:25:02.538 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33499.log 2024-04-24T01:25:02.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51292.log 2024-04-24T01:25:02.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78780.log 2024-04-24T01:25:02.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33499.log: /var/log/ceph/cluster2-client.mirror.51292.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103456.log 2024-04-24T01:25:02.546 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131558.log 2024-04-24T01:25:02.549 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.103456.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103456.log.gz 2024-04-24T01:25:02.552 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32472.log.gz 2024-04-24T01:25:02.552 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51292.log.gz 2024-04-24T01:25:02.553 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35731.log.gz 2024-04-24T01:25:02.554 INFO:teuthology.orchestra.run.smithi067.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-04-24T01:25:02.554 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69056.log 2024-04-24T01:25:02.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112095.log 2024-04-24T01:25:02.555 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131558.log.gz 2024-04-24T01:25:02.556 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74825.log 2024-04-24T01:25:02.557 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112095.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112095.log.gz 2024-04-24T01:25:02.557 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104889.log 2024-04-24T01:25:02.558 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74825.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63946.log.gz 2024-04-24T01:25:02.558 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79099.log 2024-04-24T01:25:02.558 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64398.log.gz 2024-04-24T01:25:02.559 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99086.log 2024-04-24T01:25:02.559 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93268.log 2024-04-24T01:25:02.560 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99086.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99086.log.gz 2024-04-24T01:25:02.560 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95652.log 2024-04-24T01:25:02.560 INFO:teuthology.orchestra.run.smithi134.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104889.log.gz/var/log/ceph/cluster2-client.mirror.93268.log: 2024-04-24T01:25:02.560 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74300.log 2024-04-24T01:25:02.561 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75385.log 2024-04-24T01:25:02.563 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141177.log 2024-04-24T01:25:02.563 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115801.log 2024-04-24T01:25:02.563 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75385.log: /var/log/ceph/cluster2-client.mirror.141177.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.141177.log.gz 2024-04-24T01:25:02.564 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125340.log 2024-04-24T01:25:02.565 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73561.log 2024-04-24T01:25:02.565 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.115801.log: /var/log/ceph/cluster2-client.mirror.125340.log: 87.9% 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115801.log.gz 2024-04-24T01:25:02.565 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.125340.log.gz 2024-04-24T01:25:02.565 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46673.log 2024-04-24T01:25:02.565 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73561.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125478.log 2024-04-24T01:25:02.566 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90548.log 2024-04-24T01:25:02.567 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.46673.log: /var/log/ceph/cluster2-client.mirror.125478.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125478.log.gz 2024-04-24T01:25:02.568 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126773.log 2024-04-24T01:25:02.568 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92473.log 2024-04-24T01:25:02.569 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90548.log: /var/log/ceph/cluster1-client.mirror.126773.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57973.log 2024-04-24T01:25:02.569 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92473.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126773.log.gz 2024-04-24T01:25:02.570 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104167.log 2024-04-24T01:25:02.570 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102231.log 2024-04-24T01:25:02.572 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57973.log: /var/log/ceph/cluster2-client.mirror.104167.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104167.log.gz 2024-04-24T01:25:02.574 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59518.log.gz 2024-04-24T01:25:02.574 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59553.log.gz 2024-04-24T01:25:02.574 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54206.log 2024-04-24T01:25:02.575 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.102231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46770.log 2024-04-24T01:25:02.575 INFO:teuthology.orchestra.run.smithi134.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102231.log.gz 2024-04-24T01:25:02.575 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75245.log 2024-04-24T01:25:02.576 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54206.log: /var/log/ceph/cluster2-client.mirror.46770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73246.log 2024-04-24T01:25:02.577 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51971.log 2024-04-24T01:25:02.577 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122293.log 2024-04-24T01:25:02.577 INFO:teuthology.orchestra.run.smithi134.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.98243.log.gz 2024-04-24T01:25:02.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86969.log 2024-04-24T01:25:02.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137017.log 2024-04-24T01:25:02.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.86969.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107587.log 2024-04-24T01:25:02.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137017.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137017.log.gz 2024-04-24T01:25:02.580 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47608.log 2024-04-24T01:25:02.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107587.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107587.log.gz 2024-04-24T01:25:02.580 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34929.log 2024-04-24T01:25:02.581 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47608.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34150.log.gz 2024-04-24T01:25:02.582 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95198.log.gz 2024-04-24T01:25:02.582 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129204.log 2024-04-24T01:25:02.583 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91213.log 2024-04-24T01:25:02.584 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129204.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129204.log.gz 2024-04-24T01:25:02.584 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97674.log 2024-04-24T01:25:02.584 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53077.log 2024-04-24T01:25:02.585 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.97674.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141639.log 2024-04-24T01:25:02.586 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60569.log 2024-04-24T01:25:02.586 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141639.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141639.log.gz 2024-04-24T01:25:02.586 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140752.log 2024-04-24T01:25:02.587 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40802.log 2024-04-24T01:25:02.588 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140752.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140752.log.gz 2024-04-24T01:25:02.589 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51036.log 2024-04-24T01:25:02.590 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120740.log 2024-04-24T01:25:02.590 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51036.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62484.log.gz 2024-04-24T01:25:02.595 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69056.log.gz 2024-04-24T01:25:02.596 INFO:teuthology.orchestra.run.smithi134.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.101246.log.gz 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53333.log 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120740.log: /var/log/ceph/cluster2-client.mirror.53333.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120740.log.gz 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94283.log 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106866.log 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88476.log 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124770.log 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.88476.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108142.log 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.106866.log.gz 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.124770.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95950.log.gz 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134361.log 2024-04-24T01:25:02.600 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124770.log.gz 2024-04-24T01:25:02.602 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.108142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110212.log 2024-04-24T01:25:02.602 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134361.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108142.log.gz 2024-04-24T01:25:02.602 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95718.log 2024-04-24T01:25:02.602 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137129.log 2024-04-24T01:25:02.602 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110212.log: /var/log/ceph/cluster2-client.mirror.95718.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110212.log.gz 2024-04-24T01:25:02.602 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55176.log 2024-04-24T01:25:02.604 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34464.log 2024-04-24T01:25:02.604 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.134361.log.gz 2024-04-24T01:25:02.604 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137129.log: /var/log/ceph/cluster2-client.mirror.55176.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137129.log.gz 2024-04-24T01:25:02.604 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82564.log 2024-04-24T01:25:02.604 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.34464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71916.log 2024-04-24T01:25:02.604 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39056.log 2024-04-24T01:25:02.605 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35580.log 2024-04-24T01:25:02.605 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.39056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53756.log 2024-04-24T01:25:02.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110444.log 2024-04-24T01:25:02.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54241.log 2024-04-24T01:25:02.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117559.log 2024-04-24T01:25:02.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49872.log 2024-04-24T01:25:02.606 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90793.log 2024-04-24T01:25:02.607 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117559.log: /var/log/ceph/cluster2-client.mirror.49872.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117559.log.gz 2024-04-24T01:25:02.607 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113154.log 2024-04-24T01:25:02.607 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.90793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129103.log 2024-04-24T01:25:02.608 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113154.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113154.log.gz 2024-04-24T01:25:02.608 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137422.log 2024-04-24T01:25:02.609 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129103.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.129103.log.gz 2024-04-24T01:25:02.610 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119486.log 2024-04-24T01:25:02.611 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.137422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91003.log 2024-04-24T01:25:02.611 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137422.log.gz 2024-04-24T01:25:02.612 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.119486.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119486.log.gz 2024-04-24T01:25:02.612 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80464.log 2024-04-24T01:25:02.612 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80639.log.gz 2024-04-24T01:25:02.613 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91003.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80906.log.gz 2024-04-24T01:25:02.613 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79484.log 2024-04-24T01:25:02.613 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84063.log 2024-04-24T01:25:02.614 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79484.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.30298.log.gz 2024-04-24T01:25:02.614 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48382.log 2024-04-24T01:25:02.614 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84063.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54241.log.gz 2024-04-24T01:25:02.614 INFO:teuthology.orchestra.run.smithi134.stderr:gzip 88.2% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.54206.log.gz --verbose 2024-04-24T01:25:02.614 INFO:teuthology.orchestra.run.smithi134.stderr: -- /var/log/ceph/cluster2-client.mirror.61234.log 2024-04-24T01:25:02.615 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48382.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54435.log.gz 2024-04-24T01:25:02.615 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33499.log.gz 2024-04-24T01:25:02.618 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91703.log.gz 2024-04-24T01:25:02.618 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85974.log 2024-04-24T01:25:02.619 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120296.log 2024-04-24T01:25:02.620 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94746.log 2024-04-24T01:25:02.621 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120296.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120296.log.gz 2024-04-24T01:25:02.621 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47511.log.gz 2024-04-24T01:25:02.621 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56243.log 2024-04-24T01:25:02.621 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95327.log 2024-04-24T01:25:02.622 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32693.log 2024-04-24T01:25:02.622 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56243.log: /var/log/ceph/cluster2-client.mirror.95327.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75757.log.gz 2024-04-24T01:25:02.623 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80324.log.gz 2024-04-24T01:25:02.623 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35545.log 2024-04-24T01:25:02.624 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32693.log: /var/log/ceph/cluster2-client.mirror.35545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33065.log 2024-04-24T01:25:02.625 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78780.log.gz 2024-04-24T01:25:02.625 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79484.log.gz 2024-04-24T01:25:02.625 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131280.log 2024-04-24T01:25:02.626 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57051.log 2024-04-24T01:25:02.626 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43086.log 2024-04-24T01:25:02.627 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131280.log: /var/log/ceph/cluster2-client.mirror.57051.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.131280.log.gz 2024-04-24T01:25:02.627 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86184.log 2024-04-24T01:25:02.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.43086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75140.log 2024-04-24T01:25:02.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122916.log 2024-04-24T01:25:02.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33592.log 2024-04-24T01:25:02.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.122916.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.122916.log.gz 2024-04-24T01:25:02.630 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33592.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77625.log.gz 2024-04-24T01:25:02.631 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.87081.log.gz 2024-04-24T01:25:02.631 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94283.log.gz 2024-04-24T01:25:02.631 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32507.log 2024-04-24T01:25:02.632 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134554.log 2024-04-24T01:25:02.632 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32507.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.90793.log.gz 2024-04-24T01:25:02.632 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80973.log 2024-04-24T01:25:02.633 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59133.log 2024-04-24T01:25:02.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134554.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91003.log.gz 2024-04-24T01:25:02.634 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3%/var/log/ceph/cluster1-client.mirror.80973.log: -- replaced with /var/log/ceph/cluster2-client.mirror.134554.log.gz 2024-04-24T01:25:02.634 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56472.log.gz 2024-04-24T01:25:02.634 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54726.log 2024-04-24T01:25:02.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59133.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88476.log.gz 2024-04-24T01:25:02.634 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-04-24T01:25:02.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112165.log 2024-04-24T01:25:02.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100645.log 2024-04-24T01:25:02.636 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83923.log 2024-04-24T01:25:02.636 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.112165.log: /var/log/ceph/cluster2-client.mirror.100645.log: 84.7% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112165.log.gz 2024-04-24T01:25:02.636 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100645.log.gz 2024-04-24T01:25:02.638 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61924.log 2024-04-24T01:25:02.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100555.log 2024-04-24T01:25:02.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50939.log 2024-04-24T01:25:02.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100555.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.84875.log.gz 2024-04-24T01:25:02.641 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100555.log.gz 2024-04-24T01:25:02.641 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141663.log 2024-04-24T01:25:02.642 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123080.log 2024-04-24T01:25:02.642 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36136.log 2024-04-24T01:25:02.643 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.141663.log: /var/log/ceph/cluster2-client.mirror.123080.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.141663.log.gz 2024-04-24T01:25:02.643 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.123080.log.gz 2024-04-24T01:25:02.643 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64051.log 2024-04-24T01:25:02.643 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131749.log 2024-04-24T01:25:02.643 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36136.log: /var/log/ceph/cluster2-client.mirror.64051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87321.log 2024-04-24T01:25:02.645 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116672.log 2024-04-24T01:25:02.645 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131749.log: /var/log/ceph/cluster2-client.mirror.87321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107357.log 2024-04-24T01:25:02.645 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.116672.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131749.log.gz 2024-04-24T01:25:02.645 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116672.log.gz 2024-04-24T01:25:02.645 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111213.log 2024-04-24T01:25:02.646 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35824.log 2024-04-24T01:25:02.646 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107357.log.gz/var/log/ceph/cluster1-client.mirror.111213.log: 2024-04-24T01:25:02.647 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83337.log 2024-04-24T01:25:02.648 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35824.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111213.log.gz 2024-04-24T01:25:02.648 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92634.log 2024-04-24T01:25:02.649 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69791.log 2024-04-24T01:25:02.649 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105515.log 2024-04-24T01:25:02.650 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92634.log: /var/log/ceph/cluster2-client.mirror.69791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53853.log 2024-04-24T01:25:02.650 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30089.log 2024-04-24T01:25:02.651 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.105515.log: /var/log/ceph/cluster2-client.mirror.53853.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40511.log.gz 2024-04-24T01:25:02.653 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46673.log.gz 2024-04-24T01:25:02.653 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53333.log.gz 2024-04-24T01:25:02.653 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47608.log.gz 2024-04-24T01:25:02.654 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60569.log.gz 2024-04-24T01:25:02.654 INFO:teuthology.orchestra.run.smithi134.stderr: 95.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32379.log 2024-04-24T01:25:02.655 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60709.log.gz 2024-04-24T01:25:02.655 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.105515.log.gz 2024-04-24T01:25:02.655 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.30089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132539.log 2024-04-24T01:25:02.655 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84588.log 2024-04-24T01:25:02.656 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32379.log: /var/log/ceph/cluster2-client.mirror.132539.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.132539.log.gz 2024-04-24T01:25:02.656 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113910.log 2024-04-24T01:25:02.657 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84588.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61234.log.gz 2024-04-24T01:25:02.658 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74300.log.gz 2024-04-24T01:25:02.659 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125023.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38827.log 2024-04-24T01:25:02.659 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:02.660 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111193.log 2024-04-24T01:25:02.660 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113910.log.gz 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.80973.log.gz 2024-04-24T01:25:02.660 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:02.660 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.38827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95546.log 2024-04-24T01:25:02.660 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80464.log.gz 2024-04-24T01:25:02.662 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111193.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.111193.log.gz 2024-04-24T01:25:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58888.log 2024-04-24T01:25:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66322.log 2024-04-24T01:25:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95546.log: /var/log/ceph/cluster2-client.mirror.58888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122590.log 2024-04-24T01:25:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104864.log 2024-04-24T01:25:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.122590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96686.log 2024-04-24T01:25:02.664 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.104864.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.122590.log.gz 2024-04-24T01:25:02.664 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60674.log 2024-04-24T01:25:02.665 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95175.log 2024-04-24T01:25:02.665 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104864.log.gz 2024-04-24T01:25:02.665 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96686.log: /var/log/ceph/cluster2-client.mirror.60674.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84588.log.gz 2024-04-24T01:25:02.665 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118439.log 2024-04-24T01:25:02.666 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32379.log.gz 2024-04-24T01:25:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57876.log 2024-04-24T01:25:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.95175.log: /var/log/ceph/cluster2-client.mirror.118439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136882.log 2024-04-24T01:25:02.667 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32507.log.gz 2024-04-24T01:25:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57876.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118439.log.gz 2024-04-24T01:25:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141986.log 2024-04-24T01:25:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136882.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66781.log.gz 2024-04-24T01:25:02.675 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136882.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40864.log 2024-04-24T01:25:02.676 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:02.676 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35545.log.gz 2024-04-24T01:25:02.676 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79064.log 2024-04-24T01:25:02.676 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68531.log.gz 2024-04-24T01:25:02.676 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115589.log 2024-04-24T01:25:02.676 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141986.log: /var/log/ceph/cluster2-client.mirror.79064.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141986.log.gz 2024-04-24T01:25:02.676 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100136.log 2024-04-24T01:25:02.676 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.115589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106675.log 2024-04-24T01:25:02.676 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.100136.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115589.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122251.log 2024-04-24T01:25:02.677 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:02.677 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49352.log 2024-04-24T01:25:02.677 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106675.log: /var/log/ceph/cluster2-client.mirror.122251.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83923.log.gz 2024-04-24T01:25:02.677 INFO:teuthology.orchestra.run.smithi134.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106675.log.gz 2024-04-24T01:25:02.677 INFO:teuthology.orchestra.run.smithi134.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.122251.log.gz 2024-04-24T01:25:02.677 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95390.log 2024-04-24T01:25:02.677 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132223.log 2024-04-24T01:25:02.678 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49352.log: /var/log/ceph/cluster1-client.mirror.95390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29424.log 2024-04-24T01:25:02.678 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99270.log 2024-04-24T01:25:02.678 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132223.log: /var/log/ceph/cluster1-client.admin.29424.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29424.log.gz 2024-04-24T01:25:02.678 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.132223.log.gz 2024-04-24T01:25:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132478.log 2024-04-24T01:25:02.680 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.99270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129409.log 2024-04-24T01:25:02.684 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132478.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132478.log.gz 2024-04-24T01:25:02.684 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98925.log 2024-04-24T01:25:02.684 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.129409.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129409.log.gz 2024-04-24T01:25:02.684 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76680.log 2024-04-24T01:25:02.684 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98925.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98925.log.gz 2024-04-24T01:25:02.684 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106494.log 2024-04-24T01:25:02.688 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133597.log 2024-04-24T01:25:02.688 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106494.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106494.log.gz 2024-04-24T01:25:02.688 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71510.log 2024-04-24T01:25:02.688 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133597.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133597.log.gz 2024-04-24T01:25:02.688 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59098.log 2024-04-24T01:25:02.688 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71510.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96206.log 2024-04-24T01:25:02.692 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140513.log 2024-04-24T01:25:02.692 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96206.log: /var/log/ceph/cluster2-client.mirror.140513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140142.log 2024-04-24T01:25:02.692 INFO:teuthology.orchestra.run.smithi134.stderr: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140513.log.gz 2024-04-24T01:25:02.693 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44933.log 2024-04-24T01:25:02.693 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.140142.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140142.log.gz 2024-04-24T01:25:02.693 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.105515.log 2024-04-24T01:25:02.693 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.44933.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92263.log 2024-04-24T01:25:02.699 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.105515.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35824.log.gz 2024-04-24T01:25:02.701 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53368.log 2024-04-24T01:25:02.701 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96186.log.gz 2024-04-24T01:25:02.702 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92263.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48382.log.gz 2024-04-24T01:25:02.702 INFO:teuthology.orchestra.run.smithi134.stderr: 64.1% -- replaced with /var/log/ceph/cluster1-client.mirror.30089.log.gz 2024-04-24T01:25:02.702 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56243.log.gz 2024-04-24T01:25:02.713 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49158.log.gz 2024-04-24T01:25:02.713 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49352.log.gz 2024-04-24T01:25:02.714 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49387.log.gz 2024-04-24T01:25:02.714 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54726.log.gz 2024-04-24T01:25:02.715 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79204.log 2024-04-24T01:25:02.715 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33592.log.gz 2024-04-24T01:25:02.716 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53368.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94746.log.gz 2024-04-24T01:25:02.719 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33065.log.gz 2024-04-24T01:25:02.721 INFO:teuthology.orchestra.run.smithi134.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.105515.log.gz 2024-04-24T01:25:02.721 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140407.log 2024-04-24T01:25:02.722 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.79204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141960.log 2024-04-24T01:25:02.722 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.140407.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140407.log.gz 2024-04-24T01:25:02.723 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29399.log 2024-04-24T01:25:02.723 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141960.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141960.log.gz 2024-04-24T01:25:02.723 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130024.log 2024-04-24T01:25:02.724 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.29399.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29399.log.gz 2024-04-24T01:25:02.724 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132930.log 2024-04-24T01:25:02.725 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.130024.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130024.log.gz 2024-04-24T01:25:02.725 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61094.log 2024-04-24T01:25:02.726 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132930.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132930.log.gz 2024-04-24T01:25:02.726 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91738.log 2024-04-24T01:25:02.727 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32937.log 2024-04-24T01:25:02.728 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62064.log 2024-04-24T01:25:02.728 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78010.log 2024-04-24T01:25:02.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59238.log 2024-04-24T01:25:02.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78010.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96920.log.gz 2024-04-24T01:25:02.730 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36229.log 2024-04-24T01:25:02.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114105.log 2024-04-24T01:25:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.36229.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75834.log.gz 2024-04-24T01:25:02.738 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32844.log 2024-04-24T01:25:02.739 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.114105.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114105.log.gz 2024-04-24T01:25:02.740 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104025.log 2024-04-24T01:25:02.741 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.32844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134716.log 2024-04-24T01:25:02.741 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.104025.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104025.log.gz 2024-04-24T01:25:02.742 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113119.log 2024-04-24T01:25:02.743 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134716.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134716.log.gz 2024-04-24T01:25:02.743 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.29691.log 2024-04-24T01:25:02.744 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113119.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113119.log.gz 2024-04-24T01:25:02.744 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86849.log 2024-04-24T01:25:02.745 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.2.29691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109142.log 2024-04-24T01:25:02.747 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.86849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80219.log 2024-04-24T01:25:02.749 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.109142.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79204.log.gz 2024-04-24T01:25:02.750 INFO:teuthology.orchestra.run.smithi134.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109142.log.gz 2024-04-24T01:25:02.750 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59658.log.gz 2024-04-24T01:25:02.751 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78010.log.gz 2024-04-24T01:25:02.754 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127226.log 2024-04-24T01:25:02.755 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80219.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45612.log.gz 2024-04-24T01:25:02.755 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52006.log 2024-04-24T01:25:02.756 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127226.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.127226.log.gz 2024-04-24T01:25:02.756 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69686.log 2024-04-24T01:25:02.757 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94442.log 2024-04-24T01:25:02.758 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69686.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56793.log.gz 2024-04-24T01:25:02.758 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127905.log 2024-04-24T01:25:02.758 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56913.log.gz 2024-04-24T01:25:02.759 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.94442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78535.log 2024-04-24T01:25:02.759 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127905.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127905.log.gz 2024-04-24T01:25:02.759 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111394.log 2024-04-24T01:25:02.760 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118471.log 2024-04-24T01:25:02.761 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111394.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111394.log.gz 2024-04-24T01:25:02.761 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98593.log 2024-04-24T01:25:02.762 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.118471.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118471.log.gz 2024-04-24T01:25:02.762 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121283.log 2024-04-24T01:25:02.763 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98593.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98593.log.gz 2024-04-24T01:25:02.763 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82459.log 2024-04-24T01:25:02.764 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.121283.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.121283.log.gz 2024-04-24T01:25:02.767 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60044.log 2024-04-24T01:25:02.768 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124200.log 2024-04-24T01:25:02.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60044.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.89320.log.gz 2024-04-24T01:25:02.772 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57876.log.gz 2024-04-24T01:25:02.774 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37892.log 2024-04-24T01:25:02.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.124200.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124200.log.gz 2024-04-24T01:25:02.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132365.log 2024-04-24T01:25:02.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.37892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35394.log 2024-04-24T01:25:02.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132365.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96686.log.gz 2024-04-24T01:25:02.778 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.132365.log.gz 2024-04-24T01:25:02.779 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50031.log 2024-04-24T01:25:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.35394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117644.log 2024-04-24T01:25:02.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50710.log 2024-04-24T01:25:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117644.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34929.log.gz 2024-04-24T01:25:02.782 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117644.log.gz 2024-04-24T01:25:02.782 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41578.log.gz 2024-04-24T01:25:02.783 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53077.log.gz 2024-04-24T01:25:02.784 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101802.log 2024-04-24T01:25:02.784 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50710.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53368.log.gz 2024-04-24T01:25:02.785 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60674.log.gz 2024-04-24T01:25:02.785 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60884.log.gz 2024-04-24T01:25:02.785 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61094.log.gz 2024-04-24T01:25:02.786 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73596.log.gz 2024-04-24T01:25:02.786 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69266.log 2024-04-24T01:25:02.787 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.101802.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101802.log.gz 2024-04-24T01:25:02.787 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74930.log 2024-04-24T01:25:02.788 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87181.log 2024-04-24T01:25:02.789 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74755.log.gz 2024-04-24T01:25:02.789 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29963.log 2024-04-24T01:25:02.789 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74825.log.gz 2024-04-24T01:25:02.789 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80219.log.gz 2024-04-24T01:25:02.789 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32844.log.gz 2024-04-24T01:25:02.790 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49872.log.gz 2024-04-24T01:25:02.790 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50357.log.gz 2024-04-24T01:25:02.790 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.87181.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50613.log.gz 2024-04-24T01:25:02.791 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50710.log.gz 2024-04-24T01:25:02.792 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56959.log 2024-04-24T01:25:02.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.29963.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29963.log.gz 2024-04-24T01:25:02.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96609.log 2024-04-24T01:25:02.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70845.log 2024-04-24T01:25:02.794 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.96609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45806.log 2024-04-24T01:25:02.795 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126262.log 2024-04-24T01:25:02.796 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.45806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128082.log 2024-04-24T01:25:02.797 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126262.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.126262.log.gz 2024-04-24T01:25:02.797 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83561.log.gz 2024-04-24T01:25:02.798 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81514.log 2024-04-24T01:25:02.798 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57973.log.gz 2024-04-24T01:25:02.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.128082.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128082.log.gz 2024-04-24T01:25:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137508.log 2024-04-24T01:25:02.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141687.log 2024-04-24T01:25:02.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.137508.log: 55.3% -- replaced with /var/log/ceph/cluster2-client.mirror.137508.log.gz 2024-04-24T01:25:02.800 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66532.log.gz 2024-04-24T01:25:02.802 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113631.log 2024-04-24T01:25:02.803 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35638.log.gz 2024-04-24T01:25:02.803 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.141687.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141687.log.gz 2024-04-24T01:25:02.803 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29624.log 2024-04-24T01:25:02.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.113631.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113631.log.gz 2024-04-24T01:25:02.804 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111677.log 2024-04-24T01:25:02.805 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.29624.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29624.log.gz 2024-04-24T01:25:02.806 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131245.log 2024-04-24T01:25:02.806 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111677.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111677.log.gz 2024-04-24T01:25:02.810 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132717.log 2024-04-24T01:25:02.811 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.131245.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131245.log.gz 2024-04-24T01:25:02.812 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120775.log 2024-04-24T01:25:02.812 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.132717.log: 71.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.29691.log.gz 2024-04-24T01:25:02.813 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132717.log.gz 2024-04-24T01:25:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133023.log 2024-04-24T01:25:02.815 INFO:teuthology.orchestra.run.smithi134.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.98062.log.gz 2024-04-24T01:25:02.815 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95908.log.gz 2024-04-24T01:25:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120775.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.120775.log.gz 2024-04-24T01:25:02.816 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95026.log.gz 2024-04-24T01:25:02.817 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134929.log 2024-04-24T01:25:02.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.133023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64156.log 2024-04-24T01:25:02.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134929.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133023.log.gz 2024-04-24T01:25:02.818 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134929.log.gz 2024-04-24T01:25:02.818 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98015.log 2024-04-24T01:25:02.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64156.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132646.log 2024-04-24T01:25:02.819 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49449.log 2024-04-24T01:25:02.819 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.98015.log: /var/log/ceph/cluster2-client.mirror.132646.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132646.log.gz 2024-04-24T01:25:02.820 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69266.log.gz 2024-04-24T01:25:02.820 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126390.log 2024-04-24T01:25:02.821 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49449.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69686.log.gz 2024-04-24T01:25:02.821 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74230.log 2024-04-24T01:25:02.821 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.126390.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126390.log.gz 2024-04-24T01:25:02.822 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52592.log.gz 2024-04-24T01:25:02.822 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42354.log 2024-04-24T01:25:02.823 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95112.log 2024-04-24T01:25:02.823 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.42354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33778.log 2024-04-24T01:25:02.824 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.95112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135802.log 2024-04-24T01:25:02.825 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.33778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-04-24T01:25:02.825 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.135802.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135802.log.gz 2024-04-24T01:25:02.826 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56181.log 2024-04-24T01:25:02.826 INFO:teuthology.orchestra.run.smithi134.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101727.log.gz 2024-04-24T01:25:02.827 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65321.log 2024-04-24T01:25:02.827 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39153.log 2024-04-24T01:25:02.828 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117745.log 2024-04-24T01:25:02.828 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65321.log: /var/log/ceph/cluster2-client.mirror.39153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137264.log 2024-04-24T01:25:02.828 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127762.log 2024-04-24T01:25:02.829 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.117745.log: /var/log/ceph/cluster1-client.mirror.137264.log: 88.3% 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137264.log.gz 2024-04-24T01:25:02.829 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.117745.log.gz 2024-04-24T01:25:02.829 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138539.log 2024-04-24T01:25:02.830 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.127762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136589.log 2024-04-24T01:25:02.830 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127762.log.gz 2024-04-24T01:25:02.840 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.138539.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51971.log.gz 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72966.log 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136589.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54885.log.gz 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136589.log.gz 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.138539.log.gz 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110297.log 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64531.log 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49449.log.gz 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.110297.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55405.log.gz 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110297.log.gz 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71986.log.gz 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82039.log 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6%/var/log/ceph/cluster2-client.mirror.64531.log: -- replaced with /var/log/ceph/cluster2-client.mirror.122293.log.gz 2024-04-24T01:25:02.841 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86289.log 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75140.log.gz 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109097.log 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82039.log: /var/log/ceph/cluster2-client.mirror.86289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120430.log 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96164.log.gz 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.120430.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75245.log.gz 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52456.log 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120430.log.gz 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47938.log 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.109097.log: /var/log/ceph/cluster2-client.mirror.52456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97357.log 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77520.log 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109097.log.gz/var/log/ceph/cluster2-client.mirror.97357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55564.log 2024-04-24T01:25:02.842 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:25:02.843 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77520.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20461.log 2024-04-24T01:25:02.843 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118665.log 2024-04-24T01:25:02.843 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.tmp-client.admin.20461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47449.log 2024-04-24T01:25:02.843 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20461.log.gz 2024-04-24T01:25:02.843 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.118665.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91213.log.gz 2024-04-24T01:25:02.843 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140077.log 2024-04-24T01:25:02.843 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92739.log 2024-04-24T01:25:02.843 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118665.log.gz 2024-04-24T01:25:02.844 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.47449.log: /var/log/ceph/cluster1-client.mirror.140077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55793.log 2024-04-24T01:25:02.844 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.140077.log.gz 2024-04-24T01:25:02.844 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.92739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131439.log 2024-04-24T01:25:02.844 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80149.log 2024-04-24T01:25:02.844 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55793.log: /var/log/ceph/cluster1-client.mirror.131439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141569.log 2024-04-24T01:25:02.845 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80149.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.131439.log.gz 2024-04-24T01:25:02.845 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.141569.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.141569.log.gz 2024-04-24T01:25:02.846 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107676.log 2024-04-24T01:25:02.846 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91423.log 2024-04-24T01:25:02.846 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.107676.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107676.log.gz 2024-04-24T01:25:02.847 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119073.log 2024-04-24T01:25:02.847 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.91423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.94511.log 2024-04-24T01:25:02.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.119073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139743.log 2024-04-24T01:25:02.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.94511.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.119073.log.gz 2024-04-24T01:25:02.849 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134319.log 2024-04-24T01:25:02.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.139743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40767.log 2024-04-24T01:25:02.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.134319.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.139743.log.gz 2024-04-24T01:25:02.849 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134319.log.gz 2024-04-24T01:25:02.850 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136184.log 2024-04-24T01:25:02.851 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.40767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58993.log 2024-04-24T01:25:02.851 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136342.log 2024-04-24T01:25:02.851 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136184.log.gz 2024-04-24T01:25:02.852 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111810.log 2024-04-24T01:25:02.852 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.136342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106050.log 2024-04-24T01:25:02.852 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136342.log.gz 2024-04-24T01:25:02.853 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.111810.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.111810.log.gz 2024-04-24T01:25:02.853 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.106050.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106050.log.gz 2024-04-24T01:25:02.859 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82039.log.gz 2024-04-24T01:25:02.861 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.98666.log.gz 2024-04-24T01:25:02.867 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81514.log.gz 2024-04-24T01:25:02.872 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33999.log.gz 2024-04-24T01:25:02.873 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94442.log.gz 2024-04-24T01:25:02.874 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.90548.log.gz 2024-04-24T01:25:02.877 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62274.log.gz 2024-04-24T01:25:02.883 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87531.log.gz 2024-04-24T01:25:02.897 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73561.log.gz 2024-04-24T01:25:02.898 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74230.log.gz 2024-04-24T01:25:02.899 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74930.log.gz 2024-04-24T01:25:02.900 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51036.log.gz 2024-04-24T01:25:02.901 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64156.log.gz 2024-04-24T01:25:02.906 INFO:teuthology.orchestra.run.smithi134.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81041.log.gz 2024-04-24T01:25:02.915 INFO:teuthology.orchestra.run.smithi134.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.96206.log.gz 2024-04-24T01:25:02.917 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36136.log.gz 2024-04-24T01:25:02.918 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39056.log.gz 2024-04-24T01:25:02.927 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110444.log.gz 2024-04-24T01:25:02.935 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61924.log.gz 2024-04-24T01:25:02.935 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75579.log.gz 2024-04-24T01:25:02.943 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.42354.log.gz 2024-04-24T01:25:02.961 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100136.log.gz 2024-04-24T01:25:02.967 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59133.log.gz 2024-04-24T01:25:02.967 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59098.log.gz 2024-04-24T01:25:02.967 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59238.log.gz 2024-04-24T01:25:02.968 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61409.log.gz 2024-04-24T01:25:02.969 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67831.log.gz 2024-04-24T01:25:02.970 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68251.log.gz 2024-04-24T01:25:02.970 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76680.log.gz 2024-04-24T01:25:02.972 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87181.log.gz 2024-04-24T01:25:02.980 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37892.log.gz 2024-04-24T01:25:02.980 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87321.log.gz 2024-04-24T01:25:02.982 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60044.log.gz 2024-04-24T01:25:02.984 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72966.log.gz 2024-04-24T01:25:02.985 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.99270.log.gz 2024-04-24T01:25:02.986 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67516.log.gz 2024-04-24T01:25:02.988 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.92473.log.gz 2024-04-24T01:25:02.988 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43086.log.gz 2024-04-24T01:25:02.989 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64051.log.gz 2024-04-24T01:25:02.990 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69581.log.gz 2024-04-24T01:25:02.998 INFO:teuthology.orchestra.run.smithi134.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.58602.log.gz 2024-04-24T01:25:03.004 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95718.log.gz 2024-04-24T01:25:03.016 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36229.log.gz 2024-04-24T01:25:03.018 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36447.log.gz 2024-04-24T01:25:03.019 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38827.log.gz 2024-04-24T01:25:03.021 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55114.log.gz 2024-04-24T01:25:03.021 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81374.log.gz 2024-04-24T01:25:03.021 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91738.log.gz 2024-04-24T01:25:03.047 INFO:teuthology.orchestra.run.smithi134.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98015.log.gz 2024-04-24T01:25:03.074 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92263.log.gz 2024-04-24T01:25:03.078 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86184.log.gz 2024-04-24T01:25:03.078 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86289.log.gz 2024-04-24T01:25:03.099 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64443.log.gz 2024-04-24T01:25:03.101 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66322.log.gz 2024-04-24T01:25:03.102 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66217.log.gz 2024-04-24T01:25:03.102 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73246.log.gz 2024-04-24T01:25:03.104 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.93268.log.gz 2024-04-24T01:25:03.104 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32937.log.gz 2024-04-24T01:25:03.105 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69511.log.gz 2024-04-24T01:25:03.115 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.95327.log.gz 2024-04-24T01:25:03.117 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95675.log.gz 2024-04-24T01:25:03.120 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47449.log.gz 2024-04-24T01:25:03.122 INFO:teuthology.orchestra.run.smithi134.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29652.log.gz 2024-04-24T01:25:03.127 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52006.log.gz 2024-04-24T01:25:03.132 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91423.log.gz 2024-04-24T01:25:03.134 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95390.log.gz 2024-04-24T01:25:03.137 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45806.log.gz 2024-04-24T01:25:03.170 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79064.log.gz 2024-04-24T01:25:03.171 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82459.log.gz 2024-04-24T01:25:03.173 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82564.log.gz 2024-04-24T01:25:03.174 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82879.log.gz 2024-04-24T01:25:03.177 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71510.log.gz 2024-04-24T01:25:03.178 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78150.log.gz 2024-04-24T01:25:03.181 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.90863.log.gz 2024-04-24T01:25:03.182 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58993.log.gz 2024-04-24T01:25:03.183 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67901.log.gz 2024-04-24T01:25:03.194 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40802.log.gz 2024-04-24T01:25:03.195 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40864.log.gz 2024-04-24T01:25:03.195 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53042.log.gz 2024-04-24T01:25:03.196 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47740.log.gz 2024-04-24T01:25:03.199 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32972.log.gz 2024-04-24T01:25:03.203 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50031.log.gz 2024-04-24T01:25:03.203 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35394.log.gz 2024-04-24T01:25:03.219 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55176.log.gz 2024-04-24T01:25:03.221 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55564.log.gz 2024-04-24T01:25:03.222 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56181.log.gz 2024-04-24T01:25:03.225 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94115.log.gz 2024-04-24T01:25:03.229 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33778.log.gz 2024-04-24T01:25:03.231 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53756.log.gz 2024-04-24T01:25:03.233 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55793.log.gz 2024-04-24T01:25:03.275 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.80998.log.gz 2024-04-24T01:25:03.309 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78535.log.gz 2024-04-24T01:25:03.309 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71916.log.gz 2024-04-24T01:25:03.309 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.57273.log.gz 2024-04-24T01:25:03.319 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44933.log.gz 2024-04-24T01:25:03.322 INFO:teuthology.orchestra.run.smithi134.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92739.log.gz 2024-04-24T01:25:03.328 INFO:teuthology.orchestra.run.smithi134.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.96434.log.gz 2024-04-24T01:25:03.329 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50939.log.gz 2024-04-24T01:25:03.331 INFO:teuthology.orchestra.run.smithi134.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.92634.log.gz 2024-04-24T01:25:03.346 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56959.log.gz 2024-04-24T01:25:03.346 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62064.log.gz 2024-04-24T01:25:03.346 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85974.log.gz 2024-04-24T01:25:03.369 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77520.log.gz 2024-04-24T01:25:03.404 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70845.log.gz 2024-04-24T01:25:03.411 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36560.log.gz 2024-04-24T01:25:03.423 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79099.log.gz 2024-04-24T01:25:03.424 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83337.log.gz 2024-04-24T01:25:03.424 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.83518.log.gz 2024-04-24T01:25:03.447 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84063.log.gz 2024-04-24T01:25:03.452 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.96143.log.gz 2024-04-24T01:25:03.474 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32693.log.gz 2024-04-24T01:25:03.476 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35580.log.gz 2024-04-24T01:25:03.477 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81829.log.gz 2024-04-24T01:25:03.480 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57779.log.gz 2024-04-24T01:25:03.490 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34278.log.gz 2024-04-24T01:25:03.490 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34464.log.gz 2024-04-24T01:25:03.492 INFO:teuthology.orchestra.run.smithi134.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35301.log.gz 2024-04-24T01:25:03.510 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95652.log.gz 2024-04-24T01:25:03.514 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86849.log.gz 2024-04-24T01:25:03.533 INFO:teuthology.orchestra.run.smithi134.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71784.log.gz 2024-04-24T01:25:03.534 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95546.log.gz 2024-04-24T01:25:03.588 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87846.log.gz 2024-04-24T01:25:03.609 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.93058.log.gz 2024-04-24T01:25:03.616 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62805.log.gz 2024-04-24T01:25:03.617 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80149.log.gz 2024-04-24T01:25:03.629 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69791.log.gz 2024-04-24T01:25:03.634 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.84918.log.gz 2024-04-24T01:25:03.639 INFO:teuthology.orchestra.run.smithi134.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39153.log.gz 2024-04-24T01:25:03.650 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52456.log.gz 2024-04-24T01:25:03.653 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.95175.log.gz 2024-04-24T01:25:03.656 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53853.log.gz 2024-04-24T01:25:03.666 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75385.log.gz 2024-04-24T01:25:03.677 INFO:teuthology.orchestra.run.smithi134.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97357.log.gz 2024-04-24T01:25:03.703 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.94511.log.gz 2024-04-24T01:25:03.724 INFO:teuthology.orchestra.run.smithi134.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76890.log.gz 2024-04-24T01:25:03.733 INFO:teuthology.orchestra.run.smithi134.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.94511.log.gz 2024-04-24T01:25:03.747 INFO:teuthology.orchestra.run.smithi134.stderr: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83607.log.gz 2024-04-24T01:25:03.762 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95112.log.gz 2024-04-24T01:25:03.848 INFO:teuthology.orchestra.run.smithi134.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46770.log.gz 2024-04-24T01:25:03.936 INFO:teuthology.orchestra.run.smithi134.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65321.log.gz 2024-04-24T01:25:03.947 INFO:teuthology.orchestra.run.smithi134.stderr: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42306.log.gz 2024-04-24T01:25:03.984 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95069.log.gz 2024-04-24T01:25:04.060 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40767.log.gz 2024-04-24T01:25:04.061 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57051.log.gz 2024-04-24T01:25:04.071 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86969.log.gz 2024-04-24T01:25:04.097 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.3.29824.log.gz 2024-04-24T01:25:04.220 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94158.log.gz 2024-04-24T01:25:04.225 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.94936.log.gz 2024-04-24T01:25:04.275 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95155.log.gz 2024-04-24T01:25:04.299 INFO:teuthology.orchestra.run.smithi134.stderr: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84940.log.gz 2024-04-24T01:25:04.362 INFO:teuthology.orchestra.run.smithi134.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58888.log.gz 2024-04-24T01:25:04.430 INFO:teuthology.orchestra.run.smithi134.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47938.log.gz 2024-04-24T01:25:04.451 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.83581.log.gz 2024-04-24T01:25:04.871 INFO:teuthology.orchestra.run.smithi134.stderr: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.94180.log.gz 2024-04-24T01:25:04.931 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81018.log.gz 2024-04-24T01:25:06.044 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.30183.log.gz 2024-04-24T01:25:07.410 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81211.log.gz 2024-04-24T01:25:08.884 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87109.log.gz 2024-04-24T01:25:09.828 INFO:teuthology.orchestra.run.smithi134.stderr: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97380.log.gz 2024-04-24T01:25:11.597 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.96307.log.gz 2024-04-24T01:25:12.877 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97674.log.gz 2024-04-24T01:25:13.387 INFO:teuthology.orchestra.run.smithi134.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83630.log.gz 2024-04-24T01:25:15.241 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97155.log.gz 2024-04-24T01:25:17.784 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84963.log.gz 2024-04-24T01:25:18.528 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81106.log.gz 2024-04-24T01:25:18.991 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57332.log.gz 2024-04-24T01:25:21.245 INFO:teuthology.orchestra.run.smithi134.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96943.log.gz 2024-04-24T01:25:21.315 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85167.log.gz 2024-04-24T01:25:22.452 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83760.log.gz 2024-04-24T01:25:22.494 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97215.log.gz 2024-04-24T01:25:22.637 INFO:teuthology.orchestra.run.smithi134.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97438.log.gz 2024-04-24T01:25:23.016 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76047.log.gz 2024-04-24T01:25:23.204 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.31149.log.gz 2024-04-24T01:25:24.164 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96546.log.gz 2024-04-24T01:25:24.233 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97008.log.gz 2024-04-24T01:25:24.655 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64794.log.gz 2024-04-24T01:25:24.731 INFO:teuthology.orchestra.run.smithi134.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.97087.log.gz 2024-04-24T01:25:25.109 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97514.log.gz 2024-04-24T01:25:25.935 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96609.log.gz 2024-04-24T01:25:26.107 INFO:teuthology.orchestra.run.smithi134.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96367.log.gz 2024-04-24T01:25:27.371 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36537.log.gz 2024-04-24T01:25:28.259 INFO:teuthology.orchestra.run.smithi134.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96709.log.gz 2024-04-24T01:25:28.621 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42413.log.gz 2024-04-24T01:25:28.821 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75922.log.gz 2024-04-24T01:25:29.100 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87014.log.gz 2024-04-24T01:25:30.328 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36831.log.gz 2024-04-24T01:25:30.368 INFO:teuthology.orchestra.run.smithi134.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.29651.log.gz 2024-04-24T01:25:31.193 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31068.log.gz 2024-04-24T01:25:33.375 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57159.log.gz 2024-04-24T01:25:33.777 INFO:teuthology.orchestra.run.smithi134.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97584.log.gz 2024-04-24T01:25:33.868 INFO:teuthology.orchestra.run.smithi134.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97740.log.gz 2024-04-24T01:25:36.786 INFO:teuthology.orchestra.run.smithi134.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.96736.log.gz 2024-04-24T01:25:38.255 INFO:teuthology.orchestra.run.smithi134.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.71784.log.gz 2024-04-24T01:25:39.980 INFO:teuthology.orchestra.run.smithi134.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-04-24T01:25:40.972 INFO:teuthology.orchestra.run.smithi134.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42329.log.gz 2024-04-24T01:25:47.511 INFO:teuthology.orchestra.run.smithi134.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64531.log.gz 2024-04-24T01:25:54.074 INFO:teuthology.orchestra.run.smithi134.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.29652.log.gz 2024-04-24T01:25:54.325 INFO:teuthology.orchestra.run.smithi134.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.30183.log.gz 2024-04-24T01:26:50.341 INFO:teuthology.orchestra.run.smithi067.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-04-24T01:26:53.745 INFO:teuthology.orchestra.run.smithi067.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-04-24T01:26:54.655 INFO:teuthology.orchestra.run.smithi134.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.29724.log.gz 2024-04-24T01:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-04-24T01:27:16.621 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-24T01:27:16.621 INFO:teuthology.orchestra.run.smithi067.stderr:real 2m17.939s 2024-04-24T01:27:16.621 INFO:teuthology.orchestra.run.smithi067.stderr:user 4m11.043s 2024-04-24T01:27:16.621 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m10.237s 2024-04-24T01:30:28.329 INFO:teuthology.orchestra.run.smithi134.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-04-24T01:30:33.298 INFO:teuthology.orchestra.run.smithi134.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-04-24T01:31:07.251 INFO:teuthology.orchestra.run.smithi134.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-04-24T01:31:07.252 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T01:31:07.252 INFO:teuthology.orchestra.run.smithi134.stderr:real 6m8.557s 2024-04-24T01:31:07.252 INFO:teuthology.orchestra.run.smithi134.stderr:user 7m36.534s 2024-04-24T01:31:07.252 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m23.232s 2024-04-24T01:31:07.253 INFO:tasks.ceph:Archiving logs... 2024-04-24T01:31:07.254 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535/remote/smithi067/log 2024-04-24T01:31:07.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T01:31:31.000 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535/remote/smithi134/log 2024-04-24T01:31:31.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T01:32:06.134 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T01:32:06.148 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T01:32:06.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T01:32:06.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T01:32:06.206 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T01:32:06.218 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T01:32:06.218 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T01:32:06.220 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T01:32:06.259 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T01:32:06.260 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-04-24T01:32:06.260 INFO:teuthology.orchestra.run.smithi067.stdout:*hv01.front.sepi 44.4.53.2 2 u 227 256 377 0.157 -5.201 0.288 2024-04-24T01:32:06.260 INFO:teuthology.orchestra.run.smithi067.stdout:+hv02.front.sepi 63.231.80.2 3 u 235 256 377 0.187 -3.364 0.460 2024-04-24T01:32:06.260 INFO:teuthology.orchestra.run.smithi067.stdout:+hv03.front.sepi 74.6.168.72 3 u 59 256 377 0.162 -4.241 0.328 2024-04-24T01:32:06.260 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-24T01:32:06.284 INFO:teuthology.orchestra.run.smithi134.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T01:32:06.284 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================== 2024-04-24T01:32:06.284 INFO:teuthology.orchestra.run.smithi134.stdout:*hv01.front.sepi 44.4.53.2 2 u 50 64 377 0.097 -3.148 0.720 2024-04-24T01:32:06.284 INFO:teuthology.orchestra.run.smithi134.stdout:+hv02.front.sepi 63.231.80.2 3 u 46 64 377 0.097 -1.925 0.602 2024-04-24T01:32:06.285 INFO:teuthology.orchestra.run.smithi134.stdout:+hv03.front.sepi 74.6.168.72 3 u 50 64 377 0.090 -2.147 0.638 2024-04-24T01:32:06.285 INFO:teuthology.orchestra.run.smithi134.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-24T01:32:06.286 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T01:32:06.298 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T01:32:06.298 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T01:32:06.333 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T01:32:06.372 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T01:32:06.413 INFO:teuthology.task.internal:Duration was 3892.530378 seconds 2024-04-24T01:32:06.414 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T01:32:06.446 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T01:32:06.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T01:32:06.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T01:32:06.631 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T01:32:06.632 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-04-24T01:32:06.632 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T01:32:06.647 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T01:32:06.647 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T01:32:06.659 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T01:32:06.659 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T01:32:06.694 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T01:32:06.757 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T01:32:06.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T01:32:06.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T01:32:06.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T01:32:07.206 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T01:32:07.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T01:32:07.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T01:32:07.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T01:32:07.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T01:32:07.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T01:32:07.329 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-04-24T01:32:07.330 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-04-24T01:32:07.367 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T01:32:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T01:32:07.416 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T01:32:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T01:32:07.422 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T01:32:07.462 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T01:32:07.463 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535/remote/smithi067 2024-04-24T01:32:07.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T01:32:07.488 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670535/remote/smithi134 2024-04-24T01:32:07.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T01:32:07.510 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T01:32:07.511 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T01:32:07.525 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T01:32:07.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T01:32:07.570 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T01:32:07.570 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T01:32:07.598 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T01:32:07.598 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T01:32:07.601 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T01:32:07.604 INFO:teuthology.orchestra.run.smithi067.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 01:32 /home/ubuntu/cephtest 2024-04-24T01:32:07.607 INFO:teuthology.orchestra.run.smithi134.stdout: 658627 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 01:32 /home/ubuntu/cephtest 2024-04-24T01:32:07.609 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T01:32:07.640 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T01:32:07.709 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} duration: 3892.530378341675 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-24T01:32:07.709 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T01:32:07.808 INFO:teuthology.run:pass